2024-03-28T12:09:25.460 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-28T12:09:25.460 DEBUG:teuthology.run:Teuthology command: teuthology --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} --name yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi --verbose --owner scheduled_yingxin@teuthology --archive /home/teuthworker/archive/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627765 -- /home/teuthworker/archive/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627765/orig.config.yaml 2024-03-28T12:09:25.504 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T12:09:25.586 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627765 branch: ci-yingxin-crimson-load-obc-3 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} email: yingxin.cheng@intel.com first_in_suite: false job_id: '7627765' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: ci-yingxin-crimson-load-obc-3 ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: ci-yingxin-crimson-load-obc-3 env: CRIMSON_COMPAT: '1' sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 owner: scheduled_yingxin@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 sleep_before_teardown: 0 suite: crimson-rados suite_branch: ci-yingxin-crimson-load-obc-3 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 targets: smithi183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGyf8yHXV/NPajLaUjsXrgBQJSrC9HHbq5O13zPEu3G6a09Xdx4y7dcnGZRoAn+B4cZgxQIsBsaXlkdJ6/oyt88= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4194304 pool_monitoring_list: - collectl pool_profile: replicated run_monitoring_list: - collectl time: 60 write_only: true cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-28_05:47:02 tube: smithi user: yingxin verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-28T12:09:25.586 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa; will attempt to use it 2024-03-28T12:09:25.586 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa/tasks 2024-03-28T12:09:25.587 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-28T12:09:25.588 INFO:teuthology.task.internal:Checking packages... 2024-03-28T12:09:25.609 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '785fc5e8d093374f751c7da3e51e9b56b3718d17' 2024-03-28T12:09:25.610 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-28T12:09:25.610 INFO:teuthology.packaging:ref: None 2024-03-28T12:09:25.610 INFO:teuthology.packaging:tag: None 2024-03-28T12:09:25.610 INFO:teuthology.packaging:branch: ci-yingxin-crimson-load-obc-3 2024-03-28T12:09:25.610 INFO:teuthology.packaging:sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 2024-03-28T12:09:25.610 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=ci-yingxin-crimson-load-obc-3 2024-03-28T12:09:25.823 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2473.g785fc5e8 2024-03-28T12:09:25.825 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-28T12:09:25.831 INFO:teuthology.task.internal:no buildpackages task found 2024-03-28T12:09:25.831 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-28T12:09:25.850 INFO:teuthology.task.internal:Saving configuration 2024-03-28T12:09:25.865 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-28T12:09:25.870 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-28T12:09:25.892 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627765', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-28 12:04:27.145077', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGyf8yHXV/NPajLaUjsXrgBQJSrC9HHbq5O13zPEu3G6a09Xdx4y7dcnGZRoAn+B4cZgxQIsBsaXlkdJ6/oyt88='} 2024-03-28T12:09:25.892 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-28T12:09:25.897 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-03-28T12:09:25.898 INFO:teuthology.run_tasks:Running task console_log... 2024-03-28T12:09:25.937 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8fe7ed3f70>, signals=[15]) 2024-03-28T12:09:25.937 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-28T12:09:25.942 INFO:teuthology.task.internal:Opening connections... 2024-03-28T12:09:25.942 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:09:25.944 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:09:26.032 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-28T12:09:26.038 DEBUG:teuthology.orchestra.run.smithi183:> uname -m 2024-03-28T12:09:26.103 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2024-03-28T12:09:26.103 DEBUG:teuthology.orchestra.run.smithi183:> cat /etc/os-release 2024-03-28T12:09:26.159 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="CentOS Stream" 2024-03-28T12:09:26.159 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="9" 2024-03-28T12:09:26.159 INFO:teuthology.orchestra.run.smithi183.stdout:ID="centos" 2024-03-28T12:09:26.159 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE="rhel fedora" 2024-03-28T12:09:26.159 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="9" 2024-03-28T12:09:26.159 INFO:teuthology.orchestra.run.smithi183.stdout:PLATFORM_ID="platform:el9" 2024-03-28T12:09:26.159 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-28T12:09:26.159 INFO:teuthology.orchestra.run.smithi183.stdout:ANSI_COLOR="0;31" 2024-03-28T12:09:26.160 INFO:teuthology.orchestra.run.smithi183.stdout:LOGO="fedora-logo-icon" 2024-03-28T12:09:26.160 INFO:teuthology.orchestra.run.smithi183.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-28T12:09:26.160 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://centos.org/" 2024-03-28T12:09:26.160 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-28T12:09:26.160 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-28T12:09:26.160 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-28T12:09:26.160 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2024-03-28T12:09:26.184 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-28T12:09:26.194 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-28T12:09:26.198 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-28T12:09:26.199 DEBUG:teuthology.orchestra.run.smithi183:> test '!' -e /home/ubuntu/cephtest 2024-03-28T12:09:26.215 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-28T12:09:26.220 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-28T12:09:26.220 DEBUG:teuthology.orchestra.run.smithi183:> test -z $(ls -A /var/lib/ceph) 2024-03-28T12:09:26.277 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-28T12:09:26.308 INFO:teuthology.run_tasks:Running task kernel... 2024-03-28T12:09:26.323 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-28T12:09:26.323 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-28T12:09:26.323 DEBUG:teuthology.orchestra.run.smithi183:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-28T12:09:26.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T12:09:26.338 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-03-28T12:09:26.394 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-361.el9.x86_64 2024-03-28T12:09:26.394 INFO:teuthology.task.kernel:Running kernel on smithi183: 5.14.0-361.el9.x86_64 2024-03-28T12:09:26.394 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum install -y kernel 2024-03-28T12:09:28.342 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-28T12:09:29.792 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - BaseOS 6.0 MB/s | 2.0 MB 00:00 2024-03-28T12:09:31.208 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 7.4 MB/s | 8.0 MB 00:01 2024-03-28T12:09:32.809 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 34 MB/s | 19 MB 00:00 2024-03-28T12:09:37.085 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 7.7 MB/s | 6.1 MB 00:00 2024-03-28T12:09:39.293 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 16 kB 00:01 2024-03-28T12:09:40.978 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-28T12:09:45.977 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-28T12:09:48.068 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-28T12:09:48.068 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-28T12:09:48.162 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:09:48.172 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:09:48.172 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-03-28T12:09:48.172 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:09:48.172 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-28T12:09:48.172 INFO:teuthology.orchestra.run.smithi183.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-28T12:09:48.172 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-28T12:09:48.173 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-28T12:09:48.173 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-28T12:09:48.173 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-28T12:09:48.173 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:09:48.173 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-28T12:09:48.173 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:09:48.173 INFO:teuthology.orchestra.run.smithi183.stdout:Install 4 Packages 2024-03-28T12:09:48.173 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:09:48.173 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 76 M 2024-03-28T12:09:48.174 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 123 M 2024-03-28T12:09:48.174 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-28T12:09:48.604 INFO:teuthology.orchestra.run.smithi183.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 419 kB/s | 66 kB 00:00 2024-03-28T12:09:49.729 INFO:teuthology.orchestra.run.smithi183.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 12 MB/s | 15 MB 00:01 2024-03-28T12:09:50.554 INFO:teuthology.orchestra.run.smithi183.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 16 MB/s | 33 MB 00:02 2024-03-28T12:09:52.088 INFO:teuthology.orchestra.run.smithi183.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 7.9 MB/s | 27 MB 00:03 2024-03-28T12:09:52.088 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-28T12:09:52.088 INFO:teuthology.orchestra.run.smithi183.stdout:Total 19 MB/s | 76 MB 00:03 2024-03-28T12:09:52.756 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-28T12:09:53.107 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-28T12:09:53.107 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-28T12:09:54.101 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-28T12:09:54.101 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-28T12:09:56.562 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-28T12:09:57.160 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-28T12:09:57.245 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T12:10:00.007 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T12:10:00.148 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T12:10:04.871 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T12:10:04.907 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-28T12:10:09.623 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T12:10:38.912 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T12:10:38.913 INFO:teuthology.orchestra.run.smithi183.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-28T12:10:38.913 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:10:38.956 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-28T12:10:39.309 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-28T12:10:39.310 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-28T12:10:39.310 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-28T12:10:39.310 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-28T12:10:39.696 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-28T12:10:39.697 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:10:39.697 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-28T12:10:39.697 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-28T12:10:39.697 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-28T12:10:39.697 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-28T12:10:39.697 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-28T12:10:39.697 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:10:39.697 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:10:40.007 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T12:10:40.102 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-28T12:10:40.102 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-28T12:10:40.102 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-28T12:10:40.102 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-28T12:10:40.103 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-28T12:10:40.103 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-28T12:10:40.103 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi183.front.sepia.ceph.com, path=None, version=distro) 2024-03-28T12:10:40.103 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-03-28T12:10:40.182 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-28T12:10:40.182 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-28T12:10:40.183 DEBUG:teuthology.orchestra.run.smithi183:> sudo rpm -qi grub2-tools 2024-03-28T12:10:40.216 INFO:teuthology.orchestra.run.smithi183.stdout:Name : grub2-tools 2024-03-28T12:10:40.216 INFO:teuthology.orchestra.run.smithi183.stdout:Epoch : 1 2024-03-28T12:10:40.217 INFO:teuthology.orchestra.run.smithi183.stdout:Version : 2.06 2024-03-28T12:10:40.217 INFO:teuthology.orchestra.run.smithi183.stdout:Release : 68.el9 2024-03-28T12:10:40.217 INFO:teuthology.orchestra.run.smithi183.stdout:Architecture: x86_64 2024-03-28T12:10:40.217 INFO:teuthology.orchestra.run.smithi183.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-28T12:10:40.217 INFO:teuthology.orchestra.run.smithi183.stdout:Group : Unspecified 2024-03-28T12:10:40.217 INFO:teuthology.orchestra.run.smithi183.stdout:Size : 8264254 2024-03-28T12:10:40.217 INFO:teuthology.orchestra.run.smithi183.stdout:License : GPLv3+ 2024-03-28T12:10:40.217 INFO:teuthology.orchestra.run.smithi183.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-28T12:10:40.217 INFO:teuthology.orchestra.run.smithi183.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-28T12:10:40.217 INFO:teuthology.orchestra.run.smithi183.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-28T12:10:40.217 INFO:teuthology.orchestra.run.smithi183.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-28T12:10:40.217 INFO:teuthology.orchestra.run.smithi183.stdout:Packager : builder@centos.org 2024-03-28T12:10:40.217 INFO:teuthology.orchestra.run.smithi183.stdout:Vendor : CentOS 2024-03-28T12:10:40.217 INFO:teuthology.orchestra.run.smithi183.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-28T12:10:40.217 INFO:teuthology.orchestra.run.smithi183.stdout:Summary : Support tools for GRUB. 2024-03-28T12:10:40.218 INFO:teuthology.orchestra.run.smithi183.stdout:Description : 2024-03-28T12:10:40.218 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:10:40.218 INFO:teuthology.orchestra.run.smithi183.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-28T12:10:40.218 INFO:teuthology.orchestra.run.smithi183.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-28T12:10:40.218 INFO:teuthology.orchestra.run.smithi183.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-28T12:10:40.218 INFO:teuthology.orchestra.run.smithi183.stdout:hardware devices. 2024-03-28T12:10:40.218 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:10:40.218 INFO:teuthology.orchestra.run.smithi183.stdout:This subpackage provides tools for support of all platforms. 2024-03-28T12:10:40.219 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-28T12:10:40.219 INFO:teuthology.task.kernel:Updating grub on smithi183 to boot 5.14.0-432.el9.x86_64 2024-03-28T12:10:40.219 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-28T12:10:40.786 INFO:teuthology.orchestra.run.smithi183.stderr:Generating grub configuration file ... 2024-03-28T12:10:42.163 INFO:teuthology.orchestra.run.smithi183.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-28T12:10:42.212 INFO:teuthology.orchestra.run.smithi183.stderr:done 2024-03-28T12:10:42.215 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-03-28T12:10:42.230 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.7eurxe6rqy 2024-03-28T12:10:42.230 DEBUG:teuthology.orchestra.run.smithi183:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.7eurxe6rqy 2024-03-28T12:10:42.299 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.7eurxe6rqy 2024-03-28T12:10:42.447 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.7eurxe6rqy is 6KB 2024-03-28T12:10:42.495 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.7eurxe6rqy 2024-03-28T12:10:42.509 DEBUG:teuthology.orchestra.run.smithi183:> sudo /bin/ls /boot/loader/entries || true 2024-03-28T12:10:42.574 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-28T12:10:42.575 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-28T12:10:42.575 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-28T12:10:42.575 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-28T12:10:42.576 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-28T12:10:42.735 DEBUG:teuthology.orchestra.run.smithi183:> sudo shutdown -r now 2024-03-28T12:11:12.758 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-28T12:11:12.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-28T12:11:12.759 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:12:12.764 DEBUG:teuthology.orchestra.remote:timed out 2024-03-28T12:12:21.775 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-28T12:12:21.777 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:12:22.419 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-03-28T12:12:22.843 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-28T12:12:22.843 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-28T12:12:22.843 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-03-28T12:12:22.857 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-432.el9.x86_64 2024-03-28T12:12:22.858 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-28T12:12:22.858 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-28T12:12:22.858 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-28T12:12:23.859 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-28T12:12:23.860 DEBUG:teuthology.orchestra.run.smithi183:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-28T12:12:23.923 INFO:teuthology.orchestra.run.smithi183.stdout:ttyS1 2024-03-28T12:12:23.961 DEBUG:teuthology.parallel:result is None 2024-03-28T12:12:23.962 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-28T12:12:23.970 INFO:teuthology.task.internal:Creating test directory... 2024-03-28T12:12:23.970 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-28T12:12:23.985 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-28T12:12:23.991 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-28T12:12:23.998 INFO:teuthology.task.internal:Creating archive directory... 2024-03-28T12:12:23.998 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-28T12:12:24.049 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-28T12:12:24.056 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-28T12:12:24.056 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-28T12:12:24.122 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T12:12:24.135 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-28T12:12:24.136 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-28T12:12:24.143 INFO:teuthology.task.internal:Configuring sudo... 2024-03-28T12:12:24.143 DEBUG:teuthology.orchestra.run.smithi183:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-28T12:12:24.204 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-28T12:12:24.214 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-28T12:12:24.214 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-28T12:12:24.259 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T12:12:24.341 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-28T12:12:24.395 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T12:12:24.479 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-28T12:12:24.521 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T12:12:24.521 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-28T12:12:24.589 DEBUG:teuthology.orchestra.run.smithi183:> sudo service rsyslog restart 2024-03-28T12:12:24.687 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T12:12:24.856 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-28T12:12:24.866 INFO:teuthology.task.internal:Starting timer... 2024-03-28T12:12:24.866 INFO:teuthology.run_tasks:Running task pcp... 2024-03-28T12:12:24.877 INFO:teuthology.run_tasks:Running task selinux... 2024-03-28T12:12:24.884 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-28T12:12:24.884 DEBUG:teuthology.orchestra.run.smithi183:> sudo service auditd rotate 2024-03-28T12:12:24.946 INFO:teuthology.orchestra.run.smithi183.stdout:Rotating logs: 2024-03-28T12:12:24.948 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-28T12:12:24.948 DEBUG:teuthology.orchestra.run.smithi183:> /usr/sbin/getenforce 2024-03-28T12:12:24.971 INFO:teuthology.orchestra.run.smithi183.stdout:Permissive 2024-03-28T12:12:24.972 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi183.front.sepia.ceph.com': 'permissive'} 2024-03-28T12:12:24.972 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T12:12:25.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T12:12:25.039 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-28T12:12:25.039 DEBUG:teuthology.orchestra.run.smithi183:> sudo /usr/sbin/setenforce permissive 2024-03-28T12:12:25.124 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-28T12:12:25.133 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-28T12:12:25.138 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-28T12:12:25.289 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-28T12:12:25.302 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-28T12:12:25.303 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi183.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-28T12:16:38.265 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi183.front.sepia.ceph.com')] 2024-03-28T12:16:38.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-28T12:16:38.267 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-28T12:16:38.354 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-03-28T12:16:38.436 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-28T12:16:38.437 INFO:teuthology.run_tasks:Running task clock... 2024-03-28T12:16:38.448 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-28T12:16:38.448 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-28T12:16:38.448 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T12:16:38.509 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-28T12:16:38.526 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-28T12:16:38.555 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: ntpd: command not found 2024-03-28T12:16:38.569 INFO:teuthology.orchestra.run.smithi183.stdout:506 Cannot talk to daemon 2024-03-28T12:16:38.585 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-28T12:16:38.602 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-28T12:16:38.654 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-03-28T12:16:38.657 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T12:16:38.657 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-03-28T12:16:38.657 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T12:16:38.657 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T12:16:38.657 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T12:16:38.657 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T12:16:38.658 INFO:teuthology.run_tasks:Running task install... 2024-03-28T12:16:38.666 DEBUG:teuthology.task.install:project ceph 2024-03-28T12:16:38.666 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'}} 2024-03-28T12:16:38.666 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'} 2024-03-28T12:16:38.666 INFO:teuthology.task.install:Using flavor: crimson 2024-03-28T12:16:38.672 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-28T12:16:38.672 INFO:teuthology.task.install:extra packages: [] 2024-03-28T12:16:38.673 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17', 'tag': None, 'wait_for_package': False} 2024-03-28T12:16:38.673 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=785fc5e8d093374f751c7da3e51e9b56b3718d17 2024-03-28T12:16:38.816 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-3/785fc5e8d093374f751c7da3e51e9b56b3718d17/centos/9/flavors/crimson/ 2024-03-28T12:16:38.816 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2473.g785fc5e8 2024-03-28T12:16:38.925 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-3/785fc5e8d093374f751c7da3e51e9b56b3718d17/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-3/785fc5e8d093374f751c7da3e51e9b56b3718d17/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-3/785fc5e8d093374f751c7da3e51e9b56b3718d17/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-28T12:16:38.926 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T12:16:38.926 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-28T12:16:38.953 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-28T12:16:38.954 DEBUG:teuthology.orchestra.run.smithi183:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/785fc5e8d093374f751c7da3e51e9b56b3718d17/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-28T12:16:39.022 DEBUG:teuthology.orchestra.run.smithi183:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-28T12:16:39.101 DEBUG:teuthology.orchestra.run.smithi183:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-28T12:16:39.167 INFO:teuthology.orchestra.run.smithi183.stdout:check_obsoletes = 1 2024-03-28T12:16:39.169 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum clean all 2024-03-28T12:16:39.419 INFO:teuthology.orchestra.run.smithi183.stdout:74 files removed 2024-03-28T12:16:39.449 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-radosgw 2024-03-28T12:16:40.132 INFO:teuthology.orchestra.run.smithi183.stdout:ceph packages for x86_64 226 kB/s | 87 kB 00:00 2024-03-28T12:16:40.482 INFO:teuthology.orchestra.run.smithi183.stdout:ceph noarch packages 38 kB/s | 11 kB 00:00 2024-03-28T12:16:40.849 INFO:teuthology.orchestra.run.smithi183.stdout:ceph source packages 6.2 kB/s | 2.0 kB 00:00 2024-03-28T12:16:41.335 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-28T12:16:42.952 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-03-28T12:16:44.159 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 9.3 MB/s | 8.0 MB 00:00 2024-03-28T12:16:46.760 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-28T12:16:50.902 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 6.1 MB 00:00 2024-03-28T12:16:52.578 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-03-28T12:16:54.015 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-28T12:16:58.954 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-28T12:17:01.384 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:17:01.388 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:17:01.388 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-28T12:17:01.388 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:17:01.388 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-28T12:17:01.388 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 9.6 M 2024-03-28T12:17:01.388 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrading: 2024-03-28T12:17:01.389 INFO:teuthology.orchestra.run.smithi183.stdout: librados2 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 3.4 M 2024-03-28T12:17:01.389 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 3.7 M 2024-03-28T12:17:01.389 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-28T12:17:01.389 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options 2024-03-28T12:17:01.389 INFO:teuthology.orchestra.run.smithi183.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-28T12:17:01.389 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 5.9 M 2024-03-28T12:17:01.389 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 19 M 2024-03-28T12:17:01.389 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 25 k 2024-03-28T12:17:01.389 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-28T12:17:01.389 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-28T12:17:01.389 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-28T12:17:01.389 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 796 k 2024-03-28T12:17:01.389 INFO:teuthology.orchestra.run.smithi183.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-28T12:17:01.389 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-28T12:17:01.390 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-28T12:17:01.390 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 547 k 2024-03-28T12:17:01.390 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-28T12:17:01.390 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 5.0 M 2024-03-28T12:17:01.390 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-28T12:17:01.390 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-28T12:17:01.390 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-28T12:17:01.390 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse 2024-03-28T12:17:01.390 INFO:teuthology.orchestra.run.smithi183.stdout: x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 45 k 2024-03-28T12:17:01.390 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 134 k 2024-03-28T12:17:01.390 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 159 k 2024-03-28T12:17:01.390 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-28T12:17:01.390 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 315 k 2024-03-28T12:17:01.390 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 293 k 2024-03-28T12:17:01.391 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 99 k 2024-03-28T12:17:01.391 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-28T12:17:01.391 INFO:teuthology.orchestra.run.smithi183.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-28T12:17:01.391 INFO:teuthology.orchestra.run.smithi183.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-28T12:17:01.391 INFO:teuthology.orchestra.run.smithi183.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-28T12:17:01.391 INFO:teuthology.orchestra.run.smithi183.stdout:Installing weak dependencies: 2024-03-28T12:17:01.391 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-28T12:17:01.391 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:17:01.391 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-28T12:17:01.391 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:17:01.391 INFO:teuthology.orchestra.run.smithi183.stdout:Install 30 Packages 2024-03-28T12:17:01.391 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrade 2 Packages 2024-03-28T12:17:01.391 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:17:01.392 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 58 M 2024-03-28T12:17:01.392 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-28T12:17:02.763 INFO:teuthology.orchestra.run.smithi183.stdout:(1/32): ceph-base-19.0.0-2473.g785fc5e8.el9.x86 6.6 MB/s | 5.9 MB 00:00 2024-03-28T12:17:02.830 INFO:teuthology.orchestra.run.smithi183.stdout:(2/32): ceph-selinux-19.0.0-2473.g785fc5e8.el9. 378 kB/s | 25 kB 00:00 2024-03-28T12:17:02.955 INFO:teuthology.orchestra.run.smithi183.stdout:(3/32): libcephfs2-19.0.0-2473.g785fc5e8.el9.x8 6.2 MB/s | 796 kB 00:00 2024-03-28T12:17:03.147 INFO:teuthology.orchestra.run.smithi183.stdout:(4/32): ceph-radosgw-19.0.0-2473.g785fc5e8.el9. 7.5 MB/s | 9.6 MB 00:01 2024-03-28T12:17:03.181 INFO:teuthology.orchestra.run.smithi183.stdout:(5/32): libradosstriper1-19.0.0-2473.g785fc5e8. 2.4 MB/s | 547 kB 00:00 2024-03-28T12:17:03.256 INFO:teuthology.orchestra.run.smithi183.stdout:(6/32): python3-ceph-argparse-19.0.0-2473.g785f 600 kB/s | 45 kB 00:00 2024-03-28T12:17:03.340 INFO:teuthology.orchestra.run.smithi183.stdout:(7/32): python3-ceph-common-19.0.0-2473.g785fc5 1.6 MB/s | 134 kB 00:00 2024-03-28T12:17:03.423 INFO:teuthology.orchestra.run.smithi183.stdout:(8/32): python3-cephfs-19.0.0-2473.g785fc5e8.el 1.9 MB/s | 159 kB 00:00 2024-03-28T12:17:03.515 INFO:teuthology.orchestra.run.smithi183.stdout:(9/32): python3-rados-19.0.0-2473.g785fc5e8.el9 3.4 MB/s | 315 kB 00:00 2024-03-28T12:17:03.615 INFO:teuthology.orchestra.run.smithi183.stdout:(10/32): python3-rbd-19.0.0-2473.g785fc5e8.el9. 2.9 MB/s | 293 kB 00:00 2024-03-28T12:17:03.682 INFO:teuthology.orchestra.run.smithi183.stdout:(11/32): librgw2-19.0.0-2473.g785fc5e8.el9.x86_ 9.4 MB/s | 5.0 MB 00:00 2024-03-28T12:17:03.708 INFO:teuthology.orchestra.run.smithi183.stdout:(12/32): python3-rgw-19.0.0-2473.g785fc5e8.el9. 1.1 MB/s | 99 kB 00:00 2024-03-28T12:17:03.891 INFO:teuthology.orchestra.run.smithi183.stdout:(13/32): ceph-common-19.0.0-2473.g785fc5e8.el9. 9.4 MB/s | 19 MB 00:02 2024-03-28T12:17:04.016 INFO:teuthology.orchestra.run.smithi183.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 317 kB/s | 106 kB 00:00 2024-03-28T12:17:04.041 INFO:teuthology.orchestra.run.smithi183.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 477 kB/s | 159 kB 00:00 2024-03-28T12:17:04.059 INFO:teuthology.orchestra.run.smithi183.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 370 kB/s | 44 kB 00:00 2024-03-28T12:17:04.151 INFO:teuthology.orchestra.run.smithi183.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 661 kB 00:00 2024-03-28T12:17:04.167 INFO:teuthology.orchestra.run.smithi183.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 381 kB/s | 41 kB 00:00 2024-03-28T12:17:04.201 INFO:teuthology.orchestra.run.smithi183.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 282 kB 00:00 2024-03-28T12:17:04.801 INFO:teuthology.orchestra.run.smithi183.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 57 kB/s | 36 kB 00:00 2024-03-28T12:17:04.843 INFO:teuthology.orchestra.run.smithi183.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 74 kB/s | 47 kB 00:00 2024-03-28T12:17:05.010 INFO:teuthology.orchestra.run.smithi183.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 257 kB/s | 221 kB 00:00 2024-03-28T12:17:05.102 INFO:teuthology.orchestra.run.smithi183.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.0 MB/s | 309 kB 00:00 2024-03-28T12:17:05.160 INFO:teuthology.orchestra.run.smithi183.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 831 kB/s | 49 kB 00:00 2024-03-28T12:17:05.177 INFO:teuthology.orchestra.run.smithi183.stdout:(25/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 156 kB/s | 26 kB 00:00 2024-03-28T12:17:05.227 INFO:teuthology.orchestra.run.smithi183.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-28T12:17:05.311 INFO:teuthology.orchestra.run.smithi183.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-28T12:17:05.378 INFO:teuthology.orchestra.run.smithi183.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 8.3 MB/s | 4.4 MB 00:00 2024-03-28T12:17:05.520 INFO:teuthology.orchestra.run.smithi183.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.4 MB/s | 838 kB 00:00 2024-03-28T12:17:05.562 INFO:teuthology.orchestra.run.smithi183.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 6.3 MB/s | 1.6 MB 00:00 2024-03-28T12:17:05.771 INFO:teuthology.orchestra.run.smithi183.stdout:(31/32): librados2-19.0.0-2473.g785fc5e8.el9.x8 8.7 MB/s | 3.4 MB 00:00 2024-03-28T12:17:05.879 INFO:teuthology.orchestra.run.smithi183.stdout:(32/32): librbd1-19.0.0-2473.g785fc5e8.el9.x86_ 10 MB/s | 3.7 MB 00:00 2024-03-28T12:17:05.883 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-28T12:17:05.883 INFO:teuthology.orchestra.run.smithi183.stdout:Total 13 MB/s | 58 MB 00:04 2024-03-28T12:17:05.918 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-28T12:17:05.955 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-28T12:17:05.955 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-28T12:17:06.433 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-28T12:17:06.434 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-28T12:17:07.889 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-28T12:17:08.381 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-28T12:17:08.431 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/34 2024-03-28T12:17:08.521 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/34 2024-03-28T12:17:08.610 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rados-2:19.0.0-2473.g785fc5e8.el9.x86_64 3/34 2024-03-28T12:17:08.632 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 4/34 2024-03-28T12:17:08.695 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 4/34 2024-03-28T12:17:08.738 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-03-28T12:17:08.770 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-03-28T12:17:08.804 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-03-28T12:17:08.826 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-03-28T12:17:08.843 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-03-28T12:17:09.041 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-03-28T12:17:09.076 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-03-28T12:17:09.441 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-03-28T12:17:09.472 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/34 2024-03-28T12:17:09.539 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/34 2024-03-28T12:17:09.611 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-argparse-2:19.0.0-2473.g785fc5e8.el9. 12/34 2024-03-28T12:17:09.679 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cephfs-2:19.0.0-2473.g785fc5e8.el9.x86_64 13/34 2024-03-28T12:17:09.711 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 14/34 2024-03-28T12:17:09.798 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-03-28T12:17:09.827 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libradosstriper1-2:19.0.0-2473.g785fc5e8.el9.x86_6 16/34 2024-03-28T12:17:09.912 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2473.g785fc5e8.el9.x86_6 16/34 2024-03-28T12:17:09.948 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-03-28T12:17:10.056 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-03-28T12:17:10.091 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-03-28T12:17:10.153 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-03-28T12:17:10.180 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-03-28T12:17:10.209 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-03-28T12:17:10.240 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-03-28T12:17:10.280 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-28T12:17:10.818 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-28T12:17:10.855 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librgw2-2:19.0.0-2473.g785fc5e8.el9.x86_64 26/34 2024-03-28T12:17:10.920 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librgw2-2:19.0.0-2473.g785fc5e8.el9.x86_64 26/34 2024-03-28T12:17:10.986 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 27/34 2024-03-28T12:17:11.015 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-common-2:19.0.0-2473.g785fc5e8.el9.x8 28/34 2024-03-28T12:17:13.405 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 29/34 2024-03-28T12:17:13.445 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 29/34 2024-03-28T12:17:14.125 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 29/34 2024-03-28T12:17:14.165 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 30/34 2024-03-28T12:17:14.240 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 30/34 2024-03-28T12:17:14.240 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-28T12:17:14.240 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-28T12:17:14.240 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:17:14.282 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-selinux-2:19.0.0-2473.g785fc5e8.el9.x86_64 31/34 2024-03-28T12:17:24.254 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2473.g785fc5e8.el9.x86_64 31/34 2024-03-28T12:17:24.254 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /sys 2024-03-28T12:17:24.255 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /proc 2024-03-28T12:17:24.255 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /mnt 2024-03-28T12:17:24.255 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /var/tmp 2024-03-28T12:17:24.255 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /home 2024-03-28T12:17:24.255 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /root 2024-03-28T12:17:24.255 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /tmp 2024-03-28T12:17:24.255 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:17:25.262 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-radosgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 32/34 2024-03-28T12:17:25.287 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 32/34 2024-03-28T12:17:25.287 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T12:17:25.287 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-28T12:17:25.288 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T12:17:25.288 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-28T12:17:25.288 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:17:25.290 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-28T12:17:25.365 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-28T12:17:25.366 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-28T12:17:26.561 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-28T12:17:26.561 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/34 2024-03-28T12:17:26.561 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/34 2024-03-28T12:17:26.561 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-radosgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 3/34 2024-03-28T12:17:26.561 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-selinux-2:19.0.0-2473.g785fc5e8.el9.x86_64 4/34 2024-03-28T12:17:26.561 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 5/34 2024-03-28T12:17:26.561 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libradosstriper1-2:19.0.0-2473.g785fc5e8.el9.x86_6 6/34 2024-03-28T12:17:26.561 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librgw2-2:19.0.0-2473.g785fc5e8.el9.x86_64 7/34 2024-03-28T12:17:26.561 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2473.g785fc5e8.el9. 8/34 2024-03-28T12:17:26.562 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-common-2:19.0.0-2473.g785fc5e8.el9.x8 9/34 2024-03-28T12:17:26.562 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cephfs-2:19.0.0-2473.g785fc5e8.el9.x86_64 10/34 2024-03-28T12:17:26.562 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rados-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/34 2024-03-28T12:17:26.562 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 12/34 2024-03-28T12:17:26.562 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 13/34 2024-03-28T12:17:26.562 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-28T12:17:26.562 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-28T12:17:26.562 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-28T12:17:26.562 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-28T12:17:26.562 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-28T12:17:26.562 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-28T12:17:26.562 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-28T12:17:26.562 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-28T12:17:26.563 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-28T12:17:26.563 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-28T12:17:26.563 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-28T12:17:26.563 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-28T12:17:26.563 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-28T12:17:26.563 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-28T12:17:26.563 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-28T12:17:26.563 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-28T12:17:26.563 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-28T12:17:26.563 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 31/34 2024-03-28T12:17:26.563 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-28T12:17:26.563 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 33/34 2024-03-28T12:17:27.608 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-28T12:17:27.608 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:17:27.609 INFO:teuthology.orchestra.run.smithi183.stdout:Upgraded: 2024-03-28T12:17:27.609 INFO:teuthology.orchestra.run.smithi183.stdout: librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:17:27.609 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:17:27.609 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-28T12:17:27.609 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-28T12:17:27.609 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:17:27.609 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:17:27.609 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:17:27.609 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:17:27.609 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-28T12:17:27.609 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-28T12:17:27.609 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-28T12:17:27.609 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:17:27.610 INFO:teuthology.orchestra.run.smithi183.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-28T12:17:27.610 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-28T12:17:27.610 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-28T12:17:27.610 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:17:27.610 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-28T12:17:27.610 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:17:27.610 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-28T12:17:27.610 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-28T12:17:27.610 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-28T12:17:27.610 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:17:27.610 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:17:27.610 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:17:27.610 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-28T12:17:27.610 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:17:27.611 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:17:27.611 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:17:27.611 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-28T12:17:27.611 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-28T12:17:27.611 INFO:teuthology.orchestra.run.smithi183.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-28T12:17:27.611 INFO:teuthology.orchestra.run.smithi183.stdout: re2-1:20211101-3.el9.x86_64 2024-03-28T12:17:27.611 INFO:teuthology.orchestra.run.smithi183.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-28T12:17:27.611 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:17:27.611 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:17:27.902 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-test 2024-03-28T12:17:28.458 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:30 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:17:28.610 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:17:28.612 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:17:28.612 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-28T12:17:28.612 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:17:28.612 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-28T12:17:28.612 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 87 M 2024-03-28T12:17:28.612 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-28T12:17:28.612 INFO:teuthology.orchestra.run.smithi183.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-28T12:17:28.612 INFO:teuthology.orchestra.run.smithi183.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-03-28T12:17:28.612 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 174 k 2024-03-28T12:17:28.612 INFO:teuthology.orchestra.run.smithi183.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-03-28T12:17:28.612 INFO:teuthology.orchestra.run.smithi183.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-03-28T12:17:28.612 INFO:teuthology.orchestra.run.smithi183.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-28T12:17:28.612 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-28T12:17:28.612 INFO:teuthology.orchestra.run.smithi183.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-28T12:17:28.613 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-28T12:17:28.613 INFO:teuthology.orchestra.run.smithi183.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-03-28T12:17:28.613 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:17:28.613 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-28T12:17:28.613 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:17:28.613 INFO:teuthology.orchestra.run.smithi183.stdout:Install 11 Packages 2024-03-28T12:17:28.613 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:17:28.613 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 90 M 2024-03-28T12:17:28.614 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 819 M 2024-03-28T12:17:28.614 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-28T12:17:29.220 INFO:teuthology.orchestra.run.smithi183.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-28T12:17:29.287 INFO:teuthology.orchestra.run.smithi183.stdout:(2/11): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-03-28T12:17:29.328 INFO:teuthology.orchestra.run.smithi183.stdout:(3/11): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2024-03-28T12:17:29.354 INFO:teuthology.orchestra.run.smithi183.stdout:(4/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-03-28T12:17:29.387 INFO:teuthology.orchestra.run.smithi183.stdout:(5/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 2.6 MB/s | 90 kB 00:00 2024-03-28T12:17:29.429 INFO:teuthology.orchestra.run.smithi183.stdout:(6/11): libcephsqlite-19.0.0-2473.g785fc5e8.el9 467 kB/s | 174 kB 00:00 2024-03-28T12:17:29.529 INFO:teuthology.orchestra.run.smithi183.stdout:(7/11): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-28T12:17:29.755 INFO:teuthology.orchestra.run.smithi183.stdout:(8/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 1.4 MB/s | 469 kB 00:00 2024-03-28T12:17:29.797 INFO:teuthology.orchestra.run.smithi183.stdout:(9/11): libubsan-11.4.1-3.el9.x86_64.rpm 699 kB/s | 187 kB 00:00 2024-03-28T12:17:29.964 INFO:teuthology.orchestra.run.smithi183.stdout:(10/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 583 kB/s | 122 kB 00:00 2024-03-28T12:17:33.314 INFO:teuthology.orchestra.run.smithi183.stdout:(11/11): ceph-test-19.0.0-2473.g785fc5e8.el9.x8 20 MB/s | 87 MB 00:04 2024-03-28T12:17:33.316 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-28T12:17:33.317 INFO:teuthology.orchestra.run.smithi183.stdout:Total 19 MB/s | 90 MB 00:04 2024-03-28T12:17:33.357 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-28T12:17:33.381 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-28T12:17:33.381 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-28T12:17:34.018 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-28T12:17:34.018 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-28T12:17:35.019 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-28T12:17:35.093 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-03-28T12:17:35.123 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-03-28T12:17:35.149 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-03-28T12:17:35.250 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-03-28T12:17:35.290 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-03-28T12:17:35.334 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-03-28T12:17:35.387 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-03-28T12:17:35.419 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-03-28T12:17:35.458 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-03-28T12:17:35.510 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-03-28T12:17:35.537 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephsqlite-2:19.0.0-2473.g785fc5e8.el9.x86_64 10/11 2024-03-28T12:17:47.008 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2473.g785fc5e8.el9.x86_64 10/11 2024-03-28T12:17:47.086 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-test-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/11 2024-03-28T12:17:53.176 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-test-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/11 2024-03-28T12:17:53.176 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-test-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/11 2024-03-28T12:17:53.176 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephsqlite-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/11 2024-03-28T12:17:53.176 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-03-28T12:17:53.176 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-03-28T12:17:53.176 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-03-28T12:17:53.176 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-03-28T12:17:53.176 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-03-28T12:17:53.176 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-03-28T12:17:53.176 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-03-28T12:17:53.177 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-03-28T12:17:53.626 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-03-28T12:17:53.626 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:17:53.626 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-28T12:17:53.626 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:17:53.626 INFO:teuthology.orchestra.run.smithi183.stdout: jq-1.6-16.el9.x86_64 2024-03-28T12:17:53.626 INFO:teuthology.orchestra.run.smithi183.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-03-28T12:17:53.626 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:17:53.626 INFO:teuthology.orchestra.run.smithi183.stdout: libubsan-11.4.1-3.el9.x86_64 2024-03-28T12:17:53.626 INFO:teuthology.orchestra.run.smithi183.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-03-28T12:17:53.626 INFO:teuthology.orchestra.run.smithi183.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-28T12:17:53.627 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-28T12:17:53.627 INFO:teuthology.orchestra.run.smithi183.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-28T12:17:53.627 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-28T12:17:53.627 INFO:teuthology.orchestra.run.smithi183.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-03-28T12:17:53.627 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:17:53.627 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:17:53.943 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph 2024-03-28T12:17:54.503 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:56 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:17:54.652 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:17:54.654 INFO:teuthology.orchestra.run.smithi183.stdout:========================================================================================= 2024-03-28T12:17:54.654 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-28T12:17:54.654 INFO:teuthology.orchestra.run.smithi183.stdout:========================================================================================= 2024-03-28T12:17:54.655 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-28T12:17:54.655 INFO:teuthology.orchestra.run.smithi183.stdout: ceph x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 6.5 k 2024-03-28T12:17:54.655 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-28T12:17:54.655 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 2.2 M 2024-03-28T12:17:54.655 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 1.6 M 2024-03-28T12:17:54.655 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 249 k 2024-03-28T12:17:54.655 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 5.0 M 2024-03-28T12:17:54.655 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 71 M 2024-03-28T12:17:54.655 INFO:teuthology.orchestra.run.smithi183.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-28T12:17:54.655 INFO:teuthology.orchestra.run.smithi183.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-28T12:17:54.655 INFO:teuthology.orchestra.run.smithi183.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-28T12:17:54.655 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-28T12:17:54.655 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-28T12:17:54.655 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-28T12:17:54.655 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-28T12:17:54.656 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-28T12:17:54.656 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-28T12:17:54.656 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-28T12:17:54.656 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-28T12:17:54.656 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-28T12:17:54.656 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-28T12:17:54.656 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-28T12:17:54.656 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-28T12:17:54.656 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-28T12:17:54.656 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-28T12:17:54.656 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-28T12:17:54.656 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-28T12:17:54.656 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-28T12:17:54.657 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-28T12:17:54.657 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-28T12:17:54.657 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-28T12:17:54.657 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-28T12:17:54.657 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-28T12:17:54.657 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-28T12:17:54.657 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-28T12:17:54.657 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-28T12:17:54.657 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-28T12:17:54.657 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-28T12:17:54.657 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:17:54.657 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-28T12:17:54.657 INFO:teuthology.orchestra.run.smithi183.stdout:========================================================================================= 2024-03-28T12:17:54.657 INFO:teuthology.orchestra.run.smithi183.stdout:Install 36 Packages 2024-03-28T12:17:54.657 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:17:54.658 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 84 M 2024-03-28T12:17:54.658 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 921 M 2024-03-28T12:17:54.658 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-28T12:17:55.169 INFO:teuthology.orchestra.run.smithi183.stdout:(1/36): ceph-19.0.0-2473.g785fc5e8.el9.x86_64.r 19 kB/s | 6.5 kB 00:00 2024-03-28T12:17:55.678 INFO:teuthology.orchestra.run.smithi183.stdout:(2/36): ceph-mgr-19.0.0-2473.g785fc5e8.el9.x86_ 1.8 MB/s | 1.6 MB 00:00 2024-03-28T12:17:56.662 INFO:teuthology.orchestra.run.smithi183.stdout:(3/36): ceph-mon-19.0.0-2473.g785fc5e8.el9.x86_ 3.4 MB/s | 5.0 MB 00:01 2024-03-28T12:17:56.736 INFO:teuthology.orchestra.run.smithi183.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2473.g785f 3.3 MB/s | 249 kB 00:00 2024-03-28T12:17:56.962 INFO:teuthology.orchestra.run.smithi183.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 830 kB/s | 187 kB 00:00 2024-03-28T12:17:57.037 INFO:teuthology.orchestra.run.smithi183.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-28T12:17:57.104 INFO:teuthology.orchestra.run.smithi183.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 473 kB/s | 32 kB 00:00 2024-03-28T12:17:57.171 INFO:teuthology.orchestra.run.smithi183.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 712 kB/s | 47 kB 00:00 2024-03-28T12:17:57.238 INFO:teuthology.orchestra.run.smithi183.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 579 kB/s | 39 kB 00:00 2024-03-28T12:17:57.321 INFO:teuthology.orchestra.run.smithi183.stdout:(10/36): ceph-mds-19.0.0-2473.g785fc5e8.el9.x86 895 kB/s | 2.2 MB 00:02 2024-03-28T12:17:57.364 INFO:teuthology.orchestra.run.smithi183.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.9 MB/s | 241 kB 00:00 2024-03-28T12:17:57.430 INFO:teuthology.orchestra.run.smithi183.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-28T12:17:57.457 INFO:teuthology.orchestra.run.smithi183.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-28T12:17:57.481 INFO:teuthology.orchestra.run.smithi183.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-28T12:17:57.506 INFO:teuthology.orchestra.run.smithi183.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-28T12:17:57.590 INFO:teuthology.orchestra.run.smithi183.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-03-28T12:17:57.648 INFO:teuthology.orchestra.run.smithi183.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 748 kB/s | 125 kB 00:00 2024-03-28T12:17:57.841 INFO:teuthology.orchestra.run.smithi183.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 263 kB/s | 43 kB 00:00 2024-03-28T12:17:57.866 INFO:teuthology.orchestra.run.smithi183.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 548 kB/s | 151 kB 00:00 2024-03-28T12:17:57.949 INFO:teuthology.orchestra.run.smithi183.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-03-28T12:17:57.974 INFO:teuthology.orchestra.run.smithi183.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.2 MB/s | 358 kB 00:00 2024-03-28T12:17:58.000 INFO:teuthology.orchestra.run.smithi183.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-03-28T12:17:58.025 INFO:teuthology.orchestra.run.smithi183.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2024-03-28T12:17:58.070 INFO:teuthology.orchestra.run.smithi183.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 346 kB/s | 23 kB 00:00 2024-03-28T12:17:58.084 INFO:teuthology.orchestra.run.smithi183.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 332 kB/s | 19 kB 00:00 2024-03-28T12:17:58.126 INFO:teuthology.orchestra.run.smithi183.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 334 kB/s | 20 kB 00:00 2024-03-28T12:17:58.168 INFO:teuthology.orchestra.run.smithi183.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 550 kB/s | 46 kB 00:00 2024-03-28T12:17:58.194 INFO:teuthology.orchestra.run.smithi183.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-03-28T12:17:58.227 INFO:teuthology.orchestra.run.smithi183.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.6 MB/s | 272 kB 00:00 2024-03-28T12:17:58.252 INFO:teuthology.orchestra.run.smithi183.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-03-28T12:17:58.278 INFO:teuthology.orchestra.run.smithi183.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-03-28T12:17:58.303 INFO:teuthology.orchestra.run.smithi183.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-03-28T12:17:58.337 INFO:teuthology.orchestra.run.smithi183.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.8 MB/s | 230 kB 00:00 2024-03-28T12:17:58.387 INFO:teuthology.orchestra.run.smithi183.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2024-03-28T12:17:58.412 INFO:teuthology.orchestra.run.smithi183.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-03-28T12:18:01.287 INFO:teuthology.orchestra.run.smithi183.stdout:(36/36): ceph-osd-19.0.0-2473.g785fc5e8.el9.x86 13 MB/s | 71 MB 00:05 2024-03-28T12:18:01.293 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-28T12:18:01.293 INFO:teuthology.orchestra.run.smithi183.stdout:Total 13 MB/s | 84 MB 00:06 2024-03-28T12:18:01.318 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-28T12:18:01.359 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-28T12:18:01.359 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-28T12:18:02.095 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-28T12:18:02.095 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-28T12:18:03.707 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-28T12:18:03.831 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-28T12:18:03.949 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-28T12:18:03.983 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-28T12:18:04.053 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-28T12:18:04.099 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-28T12:18:04.132 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-28T12:18:04.224 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-28T12:18:04.297 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-28T12:18:04.343 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-28T12:18:04.440 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-28T12:18:04.494 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-28T12:18:04.532 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-28T12:18:04.589 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-28T12:18:04.645 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-28T12:18:04.829 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-28T12:18:04.878 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-28T12:18:04.908 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-28T12:18:04.943 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-28T12:18:04.973 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-28T12:18:05.049 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-28T12:18:05.083 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-28T12:18:05.116 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-28T12:18:05.157 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-28T12:18:05.192 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-28T12:18:05.235 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-28T12:18:05.324 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-28T12:18:05.366 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-28T12:18:05.438 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-28T12:18:05.576 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-28T12:18:05.699 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-28T12:18:05.885 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2473.g785fc5e8.el9. 31/36 2024-03-28T12:18:05.912 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 32/36 2024-03-28T12:18:05.938 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 32/36 2024-03-28T12:18:05.939 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T12:18:05.939 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-28T12:18:05.939 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T12:18:05.939 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-28T12:18:05.939 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:16.687 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-osd-2:19.0.0-2473.g785fc5e8.el9.x86_64 33/36 2024-03-28T12:18:16.713 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-osd-2:19.0.0-2473.g785fc5e8.el9.x86_64 33/36 2024-03-28T12:18:16.713 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T12:18:16.713 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-28T12:18:16.713 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T12:18:16.713 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-28T12:18:16.714 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:17.246 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mon-2:19.0.0-2473.g785fc5e8.el9.x86_64 34/36 2024-03-28T12:18:17.269 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mon-2:19.0.0-2473.g785fc5e8.el9.x86_64 34/36 2024-03-28T12:18:17.269 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T12:18:17.269 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-28T12:18:17.269 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T12:18:17.269 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-28T12:18:17.269 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:17.504 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mds-2:19.0.0-2473.g785fc5e8.el9.x86_64 35/36 2024-03-28T12:18:17.527 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mds-2:19.0.0-2473.g785fc5e8.el9.x86_64 35/36 2024-03-28T12:18:17.527 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T12:18:17.528 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-28T12:18:17.528 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T12:18:17.528 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-28T12:18:17.528 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:17.594 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-2:19.0.0-2473.g785fc5e8.el9.x86_64 36/36 2024-03-28T12:18:24.254 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-2:19.0.0-2473.g785fc5e8.el9.x86_64 36/36 2024-03-28T12:18:24.254 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/36 2024-03-28T12:18:24.254 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mds-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/36 2024-03-28T12:18:24.255 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 3/36 2024-03-28T12:18:24.255 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mon-2:19.0.0-2473.g785fc5e8.el9.x86_64 4/36 2024-03-28T12:18:24.255 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-osd-2:19.0.0-2473.g785fc5e8.el9.x86_64 5/36 2024-03-28T12:18:24.255 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2473.g785fc5e8.el9. 6/36 2024-03-28T12:18:24.255 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-28T12:18:24.255 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-28T12:18:24.255 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-28T12:18:24.255 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-28T12:18:24.255 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-28T12:18:24.255 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-28T12:18:24.255 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-28T12:18:24.255 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-28T12:18:24.255 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-28T12:18:24.255 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-28T12:18:24.255 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-28T12:18:24.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-28T12:18:24.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-28T12:18:24.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-28T12:18:24.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-28T12:18:24.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-28T12:18:24.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-28T12:18:24.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-28T12:18:24.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-28T12:18:24.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-28T12:18:24.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-28T12:18:24.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-28T12:18:24.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-28T12:18:24.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-28T12:18:24.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-28T12:18:24.256 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-28T12:18:24.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-28T12:18:24.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-28T12:18:24.257 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-28T12:18:25.356 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-28T12:18:25.357 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:25.357 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-28T12:18:25.357 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:18:25.357 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:18:25.357 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:18:25.357 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-28T12:18:25.357 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:18:25.357 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:18:25.357 INFO:teuthology.orchestra.run.smithi183.stdout: lua-5.4.4-4.el9.x86_64 2024-03-28T12:18:25.357 INFO:teuthology.orchestra.run.smithi183.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-28T12:18:25.357 INFO:teuthology.orchestra.run.smithi183.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-28T12:18:25.357 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-28T12:18:25.357 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-28T12:18:25.357 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-28T12:18:25.358 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-28T12:18:25.358 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-28T12:18:25.358 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-28T12:18:25.358 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-28T12:18:25.358 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-28T12:18:25.358 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-28T12:18:25.358 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-28T12:18:25.358 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-28T12:18:25.358 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-28T12:18:25.358 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-28T12:18:25.358 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-28T12:18:25.358 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-28T12:18:25.358 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply-3.11-14.el9.noarch 2024-03-28T12:18:25.359 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-28T12:18:25.359 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-28T12:18:25.359 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-28T12:18:25.359 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-28T12:18:25.359 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-28T12:18:25.359 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-28T12:18:25.359 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-28T12:18:25.359 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-28T12:18:25.359 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-28T12:18:25.359 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-28T12:18:25.359 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-28T12:18:25.359 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:25.359 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:18:25.626 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-base 2024-03-28T12:18:26.189 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:28 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:18:26.286 INFO:teuthology.orchestra.run.smithi183.stdout:Package ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-28T12:18:26.338 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:18:26.339 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-28T12:18:26.339 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:18:26.411 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install cephadm 2024-03-28T12:18:26.967 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:28 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:18:27.117 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:18:27.118 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:18:27.118 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-28T12:18:27.118 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:18:27.118 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-28T12:18:27.118 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 762 k 2024-03-28T12:18:27.118 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:27.118 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-28T12:18:27.118 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:18:27.119 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-28T12:18:27.119 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:27.119 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 762 k 2024-03-28T12:18:27.119 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 768 k 2024-03-28T12:18:27.119 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-28T12:18:27.397 INFO:teuthology.orchestra.run.smithi183.stdout:cephadm-19.0.0-2473.g785fc5e8.el9.noarch.rpm 2.7 MB/s | 762 kB 00:00 2024-03-28T12:18:27.397 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-28T12:18:27.398 INFO:teuthology.orchestra.run.smithi183.stdout:Total 2.7 MB/s | 762 kB 00:00 2024-03-28T12:18:27.398 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-28T12:18:27.403 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-28T12:18:27.403 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-28T12:18:27.409 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-28T12:18:27.409 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-28T12:18:27.482 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-28T12:18:27.924 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/1 2024-03-28T12:18:28.002 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/1 2024-03-28T12:18:28.394 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/1 2024-03-28T12:18:28.670 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/1 2024-03-28T12:18:28.670 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:28.670 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-28T12:18:28.671 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-28T12:18:28.671 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:28.671 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:18:28.877 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-immutable-object-cache 2024-03-28T12:18:29.450 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:31 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:18:29.606 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:18:29.607 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:18:29.607 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-28T12:18:29.607 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:18:29.607 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-28T12:18:29.607 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 157 k 2024-03-28T12:18:29.608 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:29.608 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-28T12:18:29.608 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:18:29.608 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-28T12:18:29.608 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:29.608 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 157 k 2024-03-28T12:18:29.608 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 640 k 2024-03-28T12:18:29.608 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-28T12:18:29.837 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-immutable-object-cache-19.0.0-2473.g785fc5 686 kB/s | 157 kB 00:00 2024-03-28T12:18:29.838 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-28T12:18:29.838 INFO:teuthology.orchestra.run.smithi183.stdout:Total 682 kB/s | 157 kB 00:00 2024-03-28T12:18:29.838 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-28T12:18:29.847 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-28T12:18:29.847 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-28T12:18:29.900 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-28T12:18:29.900 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-28T12:18:30.049 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-28T12:18:30.105 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2473.g785fc5e8. 1/1 2024-03-28T12:18:30.129 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2473.g785fc5e8. 1/1 2024-03-28T12:18:30.129 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T12:18:30.129 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-28T12:18:30.130 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:31.103 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2473.g785fc5e8. 1/1 2024-03-28T12:18:31.103 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:31.103 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-28T12:18:31.103 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:18:31.103 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:31.103 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:18:31.342 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr 2024-03-28T12:18:31.919 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:33 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:18:32.019 INFO:teuthology.orchestra.run.smithi183.stdout:Package ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-28T12:18:32.071 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:18:32.072 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-28T12:18:32.072 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:18:32.140 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-dashboard 2024-03-28T12:18:32.699 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:34 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:18:32.846 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:18:32.848 INFO:teuthology.orchestra.run.smithi183.stdout:====================================================================================== 2024-03-28T12:18:32.848 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-28T12:18:32.848 INFO:teuthology.orchestra.run.smithi183.stdout:====================================================================================== 2024-03-28T12:18:32.848 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-28T12:18:32.848 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 3.5 M 2024-03-28T12:18:32.848 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-28T12:18:32.848 INFO:teuthology.orchestra.run.smithi183.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-28T12:18:32.848 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 34 k 2024-03-28T12:18:32.848 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 16 k 2024-03-28T12:18:32.848 INFO:teuthology.orchestra.run.smithi183.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-28T12:18:32.848 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-28T12:18:32.848 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-28T12:18:32.849 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-28T12:18:32.849 INFO:teuthology.orchestra.run.smithi183.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-28T12:18:32.849 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-28T12:18:32.849 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-28T12:18:32.849 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:32.849 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-28T12:18:32.849 INFO:teuthology.orchestra.run.smithi183.stdout:====================================================================================== 2024-03-28T12:18:32.849 INFO:teuthology.orchestra.run.smithi183.stdout:Install 11 Packages 2024-03-28T12:18:32.849 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:32.850 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 7.6 M 2024-03-28T12:18:32.850 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 100 M 2024-03-28T12:18:32.850 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-28T12:18:33.342 INFO:teuthology.orchestra.run.smithi183.stdout:(1/11): ceph-prometheus-alerts-19.0.0-2473.g785 93 kB/s | 16 kB 00:00 2024-03-28T12:18:33.425 INFO:teuthology.orchestra.run.smithi183.stdout:(2/11): ceph-grafana-dashboards-19.0.0-2473.g78 130 kB/s | 34 kB 00:00 2024-03-28T12:18:33.492 INFO:teuthology.orchestra.run.smithi183.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-03-28T12:18:33.676 INFO:teuthology.orchestra.run.smithi183.stdout:(4/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.9 MB/s | 548 kB 00:00 2024-03-28T12:18:33.718 INFO:teuthology.orchestra.run.smithi183.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 468 kB/s | 19 kB 00:00 2024-03-28T12:18:33.801 INFO:teuthology.orchestra.run.smithi183.stdout:(6/11): ceph-mgr-dashboard-19.0.0-2473.g785fc5e 5.6 MB/s | 3.5 MB 00:00 2024-03-28T12:18:33.876 INFO:teuthology.orchestra.run.smithi183.stdout:(7/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 13 MB/s | 2.0 MB 00:00 2024-03-28T12:18:33.918 INFO:teuthology.orchestra.run.smithi183.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 737 kB/s | 31 kB 00:00 2024-03-28T12:18:33.952 INFO:teuthology.orchestra.run.smithi183.stdout:(9/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 960 kB/s | 144 kB 00:00 2024-03-28T12:18:33.998 INFO:teuthology.orchestra.run.smithi183.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 2.3 MB/s | 188 kB 00:00 2024-03-28T12:18:34.194 INFO:teuthology.orchestra.run.smithi183.stdout:(11/11): protobuf-compiler-3.14.0-13.el9.x86_64 1.1 MB/s | 863 kB 00:00 2024-03-28T12:18:34.197 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-28T12:18:34.197 INFO:teuthology.orchestra.run.smithi183.stdout:Total 5.6 MB/s | 7.6 MB 00:01 2024-03-28T12:18:34.214 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-28T12:18:34.231 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-28T12:18:34.231 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-28T12:18:34.429 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-28T12:18:34.429 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-28T12:18:34.909 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-28T12:18:34.993 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-03-28T12:18:35.023 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-03-28T12:18:35.127 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-03-28T12:18:35.257 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-03-28T12:18:35.317 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-03-28T12:18:35.403 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-03-28T12:18:35.452 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-03-28T12:18:35.481 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-03-28T12:18:35.513 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2473.g785fc5e8.el9 9/11 2024-03-28T12:18:36.569 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2473.g785fc5e8.el 10/11 2024-03-28T12:18:36.605 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2473.g785fc5e8.el9.noa 11/11 2024-03-28T12:18:37.942 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2473.g785fc5e8.el9.noa 11/11 2024-03-28T12:18:37.942 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2473.g785fc5e8.el 1/11 2024-03-28T12:18:37.942 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2473.g785fc5e8.el9.noa 2/11 2024-03-28T12:18:37.942 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2473.g785fc5e8.el9 3/11 2024-03-28T12:18:37.942 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-03-28T12:18:37.943 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-03-28T12:18:37.943 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-03-28T12:18:37.943 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-03-28T12:18:37.943 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-03-28T12:18:37.943 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-03-28T12:18:37.943 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-03-28T12:18:38.301 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-03-28T12:18:38.301 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:38.301 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-28T12:18:38.301 INFO:teuthology.orchestra.run.smithi183.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-28T12:18:38.301 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-28T12:18:38.301 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-28T12:18:38.301 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-28T12:18:38.301 INFO:teuthology.orchestra.run.smithi183.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-28T12:18:38.301 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-28T12:18:38.301 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-28T12:18:38.301 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-28T12:18:38.301 INFO:teuthology.orchestra.run.smithi183.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-28T12:18:38.301 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-28T12:18:38.302 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-28T12:18:38.302 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:38.302 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:18:38.539 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-28T12:18:39.093 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:41 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:18:39.244 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:18:39.246 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-28T12:18:39.246 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-28T12:18:39.246 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-28T12:18:39.246 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-28T12:18:39.246 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 7.4 M 2024-03-28T12:18:39.246 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-28T12:18:39.246 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-28T12:18:39.246 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-28T12:18:39.246 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-28T12:18:39.246 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-28T12:18:39.246 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-28T12:18:39.246 INFO:teuthology.orchestra.run.smithi183.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-28T12:18:39.246 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-28T12:18:39.247 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-28T12:18:39.247 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-28T12:18:39.247 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-28T12:18:39.247 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-28T12:18:39.247 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:39.247 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-28T12:18:39.247 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-28T12:18:39.247 INFO:teuthology.orchestra.run.smithi183.stdout:Install 12 Packages 2024-03-28T12:18:39.247 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:39.247 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 37 M 2024-03-28T12:18:39.248 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 209 M 2024-03-28T12:18:39.248 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-28T12:18:39.926 INFO:teuthology.orchestra.run.smithi183.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 243 kB/s | 32 kB 00:00 2024-03-28T12:18:39.960 INFO:teuthology.orchestra.run.smithi183.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 535 kB/s | 17 kB 00:00 2024-03-28T12:18:40.018 INFO:teuthology.orchestra.run.smithi183.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-03-28T12:18:40.043 INFO:teuthology.orchestra.run.smithi183.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 419 kB/s | 34 kB 00:00 2024-03-28T12:18:40.202 INFO:teuthology.orchestra.run.smithi183.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2024-03-28T12:18:40.269 INFO:teuthology.orchestra.run.smithi183.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-28T12:18:40.295 INFO:teuthology.orchestra.run.smithi183.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-28T12:18:40.620 INFO:teuthology.orchestra.run.smithi183.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-28T12:18:40.812 INFO:teuthology.orchestra.run.smithi183.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-24 7.3 MB/s | 7.4 MB 00:01 2024-03-28T12:18:40.845 INFO:teuthology.orchestra.run.smithi183.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 803 kB 00:00 2024-03-28T12:18:40.895 INFO:teuthology.orchestra.run.smithi183.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 712 kB/s | 195 kB 00:00 2024-03-28T12:18:41.045 INFO:teuthology.orchestra.run.smithi183.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-03-28T12:18:41.049 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-28T12:18:41.049 INFO:teuthology.orchestra.run.smithi183.stdout:Total 21 MB/s | 37 MB 00:01 2024-03-28T12:18:41.089 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-28T12:18:41.113 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-28T12:18:41.113 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-28T12:18:41.467 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-28T12:18:41.467 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-28T12:18:42.191 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-28T12:18:42.268 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-28T12:18:42.299 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-28T12:18:42.400 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-28T12:18:42.444 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-28T12:18:42.622 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-28T12:18:42.652 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-28T12:18:42.734 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-28T12:18:43.373 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-28T12:18:43.481 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-28T12:18:44.714 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-28T12:18:45.734 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-28T12:18:45.768 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2473.g785fc 12/12 2024-03-28T12:18:48.226 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2473.g785fc 12/12 2024-03-28T12:18:48.226 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2473.g785fc 1/12 2024-03-28T12:18:48.226 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-28T12:18:48.226 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-28T12:18:48.226 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-28T12:18:48.226 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-28T12:18:48.226 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-28T12:18:48.226 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-28T12:18:48.227 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-28T12:18:48.227 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-28T12:18:48.227 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-28T12:18:48.227 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-28T12:18:48.710 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-28T12:18:48.711 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:48.711 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-28T12:18:48.711 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-28T12:18:48.711 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-28T12:18:48.711 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-28T12:18:48.711 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-28T12:18:48.711 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-28T12:18:48.711 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-28T12:18:48.711 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-28T12:18:48.711 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-28T12:18:48.711 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-28T12:18:48.711 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-28T12:18:48.711 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-28T12:18:48.711 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-28T12:18:48.712 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:48.712 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:18:49.188 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-rook 2024-03-28T12:18:49.775 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:51 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:18:49.937 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:18:49.938 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-28T12:18:49.938 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-28T12:18:49.938 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-28T12:18:49.938 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-28T12:18:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 49 k 2024-03-28T12:18:49.939 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-28T12:18:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-28T12:18:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-28T12:18:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-28T12:18:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-28T12:18:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-28T12:18:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-28T12:18:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-28T12:18:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-28T12:18:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-28T12:18:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-28T12:18:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-28T12:18:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-28T12:18:49.939 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:49.940 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-28T12:18:49.940 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-28T12:18:49.940 INFO:teuthology.orchestra.run.smithi183.stdout:Install 13 Packages 2024-03-28T12:18:49.940 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:49.940 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 2.0 M 2024-03-28T12:18:49.940 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 26 M 2024-03-28T12:18:49.940 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-28T12:18:50.211 INFO:teuthology.orchestra.run.smithi183.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 149 kB/s | 18 kB 00:00 2024-03-28T12:18:50.236 INFO:teuthology.orchestra.run.smithi183.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 167 kB/s | 24 kB 00:00 2024-03-28T12:18:50.286 INFO:teuthology.orchestra.run.smithi183.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-28T12:18:50.312 INFO:teuthology.orchestra.run.smithi183.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-28T12:18:50.337 INFO:teuthology.orchestra.run.smithi183.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-28T12:18:50.362 INFO:teuthology.orchestra.run.smithi183.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 858 kB/s | 43 kB 00:00 2024-03-28T12:18:50.387 INFO:teuthology.orchestra.run.smithi183.stdout:(7/13): ceph-mgr-rook-19.0.0-2473.g785fc5e8.el9 165 kB/s | 49 kB 00:00 2024-03-28T12:18:50.471 INFO:teuthology.orchestra.run.smithi183.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 138 kB/s | 15 kB 00:00 2024-03-28T12:18:50.496 INFO:teuthology.orchestra.run.smithi183.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 202 kB/s | 32 kB 00:00 2024-03-28T12:18:50.605 INFO:teuthology.orchestra.run.smithi183.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 545 kB/s | 59 kB 00:00 2024-03-28T12:18:50.630 INFO:teuthology.orchestra.run.smithi183.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 836 kB/s | 203 kB 00:00 2024-03-28T12:18:50.680 INFO:teuthology.orchestra.run.smithi183.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-03-28T12:18:50.747 INFO:teuthology.orchestra.run.smithi183.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.7 MB/s | 1.0 MB 00:00 2024-03-28T12:18:50.749 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-28T12:18:50.749 INFO:teuthology.orchestra.run.smithi183.stdout:Total 2.5 MB/s | 2.0 MB 00:00 2024-03-28T12:18:50.750 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-28T12:18:50.763 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-28T12:18:50.763 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-28T12:18:50.852 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-28T12:18:50.852 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-28T12:18:51.370 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-28T12:18:51.544 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-28T12:18:51.609 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-28T12:18:51.670 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-28T12:18:51.708 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-28T12:18:51.749 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-28T12:18:51.842 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-28T12:18:51.942 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-28T12:18:51.991 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-28T12:18:52.708 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-28T12:18:52.758 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-28T12:18:52.792 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-28T12:18:52.840 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-28T12:18:52.862 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-rook-2:19.0.0-2473.g785fc5e8.el9.noarch 13/13 2024-03-28T12:18:53.609 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2473.g785fc5e8.el9.noarch 13/13 2024-03-28T12:18:53.609 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2473.g785fc5e8.el9.noarch 1/13 2024-03-28T12:18:53.609 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-28T12:18:53.609 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-28T12:18:53.609 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-28T12:18:53.609 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-28T12:18:53.609 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-28T12:18:53.609 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-28T12:18:53.609 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-28T12:18:53.610 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-28T12:18:53.610 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-28T12:18:53.610 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-28T12:18:53.610 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-28T12:18:53.966 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-28T12:18:53.966 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:53.966 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-28T12:18:53.966 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-28T12:18:53.966 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-28T12:18:53.966 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-28T12:18:53.966 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-28T12:18:53.966 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-28T12:18:53.966 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-28T12:18:53.967 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-28T12:18:53.967 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-28T12:18:53.967 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-28T12:18:53.967 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-28T12:18:53.967 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-28T12:18:53.967 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-28T12:18:53.967 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-28T12:18:53.967 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:53.967 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:18:54.197 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-cephadm 2024-03-28T12:18:54.759 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:56 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:18:54.909 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:18:54.910 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-28T12:18:54.910 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-28T12:18:54.910 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-28T12:18:54.910 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-28T12:18:54.910 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 141 k 2024-03-28T12:18:54.911 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-28T12:18:54.911 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-28T12:18:54.911 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-28T12:18:54.911 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-28T12:18:54.911 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-28T12:18:54.911 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-28T12:18:54.911 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:54.911 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-28T12:18:54.911 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-28T12:18:54.911 INFO:teuthology.orchestra.run.smithi183.stdout:Install 6 Packages 2024-03-28T12:18:54.911 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:54.912 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 6.8 M 2024-03-28T12:18:54.912 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 33 M 2024-03-28T12:18:54.912 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-28T12:18:55.426 INFO:teuthology.orchestra.run.smithi183.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-03-28T12:18:55.510 INFO:teuthology.orchestra.run.smithi183.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-28T12:18:55.610 INFO:teuthology.orchestra.run.smithi183.stdout:(3/6): ceph-mgr-cephadm-19.0.0-2473.g785fc5e8.e 375 kB/s | 141 kB 00:00 2024-03-28T12:18:55.986 INFO:teuthology.orchestra.run.smithi183.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 128 kB/s | 58 kB 00:00 2024-03-28T12:18:56.090 INFO:teuthology.orchestra.run.smithi183.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 112 kB/s | 51 kB 00:00 2024-03-28T12:18:56.136 INFO:teuthology.orchestra.run.smithi183.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 775 kB/s | 548 kB 00:00 2024-03-28T12:18:56.140 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-28T12:18:56.140 INFO:teuthology.orchestra.run.smithi183.stdout:Total 5.5 MB/s | 6.8 MB 00:01 2024-03-28T12:18:56.152 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-28T12:18:56.163 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-28T12:18:56.163 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-28T12:18:56.251 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-28T12:18:56.251 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-28T12:18:56.542 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-28T12:18:56.725 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-28T12:18:56.779 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-28T12:18:57.291 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-28T12:18:57.382 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-28T12:18:57.445 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-28T12:18:57.483 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 6/6 2024-03-28T12:18:58.274 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 6/6 2024-03-28T12:18:58.274 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/6 2024-03-28T12:18:58.274 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-28T12:18:58.275 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-28T12:18:58.275 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-28T12:18:58.275 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-28T12:18:58.598 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-28T12:18:58.599 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:58.599 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-28T12:18:58.599 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-28T12:18:58.599 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-28T12:18:58.599 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-28T12:18:58.599 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-28T12:18:58.599 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-28T12:18:58.599 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-28T12:18:58.599 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:58.599 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:18:58.869 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-fuse 2024-03-28T12:18:59.430 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:01 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:18:59.584 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:18:59.584 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:18:59.584 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-28T12:18:59.585 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:18:59.585 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-28T12:18:59.585 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 871 k 2024-03-28T12:18:59.585 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-28T12:18:59.585 INFO:teuthology.orchestra.run.smithi183.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-28T12:18:59.585 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:59.585 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-28T12:18:59.585 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:18:59.585 INFO:teuthology.orchestra.run.smithi183.stdout:Install 2 Packages 2024-03-28T12:18:59.585 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:18:59.586 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 949 k 2024-03-28T12:18:59.586 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 4.0 M 2024-03-28T12:18:59.586 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-28T12:18:59.732 INFO:teuthology.orchestra.run.smithi183.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 537 kB/s | 79 kB 00:00 2024-03-28T12:18:59.929 INFO:teuthology.orchestra.run.smithi183.stdout:(2/2): ceph-fuse-19.0.0-2473.g785fc5e8.el9.x86_ 2.5 MB/s | 871 kB 00:00 2024-03-28T12:18:59.931 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-28T12:18:59.931 INFO:teuthology.orchestra.run.smithi183.stdout:Total 2.7 MB/s | 949 kB 00:00 2024-03-28T12:18:59.931 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-28T12:18:59.954 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-28T12:18:59.954 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-28T12:19:00.027 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-28T12:19:00.027 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-28T12:19:00.228 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-28T12:19:00.389 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-28T12:19:00.454 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/2 2024-03-28T12:19:01.176 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/2 2024-03-28T12:19:01.176 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/2 2024-03-28T12:19:01.522 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-28T12:19:01.522 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:01.523 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-28T12:19:01.523 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-28T12:19:01.523 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:01.523 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:19:01.785 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-volume 2024-03-28T12:19:02.364 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:04 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:19:02.514 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:19:02.514 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:19:02.515 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-28T12:19:02.515 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:19:02.515 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-28T12:19:02.515 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 264 k 2024-03-28T12:19:02.515 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-28T12:19:02.515 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-28T12:19:02.515 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:02.515 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-28T12:19:02.515 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:19:02.515 INFO:teuthology.orchestra.run.smithi183.stdout:Install 2 Packages 2024-03-28T12:19:02.515 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:02.516 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 332 k 2024-03-28T12:19:02.516 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 1.5 M 2024-03-28T12:19:02.516 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-28T12:19:02.634 INFO:teuthology.orchestra.run.smithi183.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 580 kB/s | 69 kB 00:00 2024-03-28T12:19:02.876 INFO:teuthology.orchestra.run.smithi183.stdout:(2/2): ceph-volume-19.0.0-2473.g785fc5e8.el9.no 732 kB/s | 264 kB 00:00 2024-03-28T12:19:02.878 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-28T12:19:02.878 INFO:teuthology.orchestra.run.smithi183.stdout:Total 917 kB/s | 332 kB 00:00 2024-03-28T12:19:02.878 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-28T12:19:02.886 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-28T12:19:02.886 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-28T12:19:02.927 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-28T12:19:02.927 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-28T12:19:03.163 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-28T12:19:03.333 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-28T12:19:03.392 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-volume-2:19.0.0-2473.g785fc5e8.el9.noarch 2/2 2024-03-28T12:19:03.409 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-volume-2:19.0.0-2473.g785fc5e8.el9.noarch 2/2 2024-03-28T12:19:03.409 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T12:19:03.409 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-28T12:19:03.409 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:04.124 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-volume-2:19.0.0-2473.g785fc5e8.el9.noarch 1/2 2024-03-28T12:19:04.499 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-28T12:19:04.499 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:04.499 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-28T12:19:04.500 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-28T12:19:04.500 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-28T12:19:04.500 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:04.500 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:19:04.779 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librados-devel 2024-03-28T12:19:05.353 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:07 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:19:05.503 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:19:05.504 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:19:05.504 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-28T12:19:05.504 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:19:05.504 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-28T12:19:05.504 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 125 k 2024-03-28T12:19:05.504 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:05.504 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-28T12:19:05.504 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:19:05.504 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-28T12:19:05.504 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:05.505 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 125 k 2024-03-28T12:19:05.505 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 445 k 2024-03-28T12:19:05.505 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-28T12:19:05.824 INFO:teuthology.orchestra.run.smithi183.stdout:librados-devel-19.0.0-2473.g785fc5e8.el9.x86_64 391 kB/s | 125 kB 00:00 2024-03-28T12:19:05.825 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-28T12:19:05.825 INFO:teuthology.orchestra.run.smithi183.stdout:Total 389 kB/s | 125 kB 00:00 2024-03-28T12:19:05.825 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-28T12:19:05.831 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-28T12:19:05.831 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-28T12:19:05.888 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-28T12:19:05.888 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-28T12:19:06.035 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-28T12:19:06.139 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librados-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-28T12:19:06.647 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-28T12:19:06.938 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-28T12:19:06.938 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:06.938 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-28T12:19:06.938 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:19:06.938 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:06.938 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:19:07.201 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install libcephfs2 2024-03-28T12:19:07.775 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:09 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:19:07.875 INFO:teuthology.orchestra.run.smithi183.stdout:Package libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-28T12:19:07.929 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:19:07.930 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-28T12:19:07.931 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:19:08.004 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install libcephfs-devel 2024-03-28T12:19:08.565 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:10 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:19:08.715 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:19:08.716 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:19:08.716 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-28T12:19:08.716 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:19:08.716 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-28T12:19:08.716 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 32 k 2024-03-28T12:19:08.716 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:08.716 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-28T12:19:08.716 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:19:08.716 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-28T12:19:08.716 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:08.717 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 32 k 2024-03-28T12:19:08.717 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 142 k 2024-03-28T12:19:08.717 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-28T12:19:08.961 INFO:teuthology.orchestra.run.smithi183.stdout:libcephfs-devel-19.0.0-2473.g785fc5e8.el9.x86_6 130 kB/s | 32 kB 00:00 2024-03-28T12:19:08.961 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-28T12:19:08.962 INFO:teuthology.orchestra.run.smithi183.stdout:Total 129 kB/s | 32 kB 00:00 2024-03-28T12:19:08.962 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-28T12:19:08.965 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-28T12:19:08.965 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-28T12:19:08.991 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-28T12:19:08.992 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-28T12:19:09.083 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-28T12:19:09.193 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-28T12:19:09.565 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-28T12:19:09.826 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-28T12:19:09.826 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:09.826 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-28T12:19:09.826 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:19:09.826 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:09.826 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:19:10.008 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librados2 2024-03-28T12:19:10.569 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:12 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:19:10.667 INFO:teuthology.orchestra.run.smithi183.stdout:Package librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-28T12:19:10.719 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:19:10.720 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-28T12:19:10.720 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:19:10.795 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librbd1 2024-03-28T12:19:11.348 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:13 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:19:11.443 INFO:teuthology.orchestra.run.smithi183.stdout:Package librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-28T12:19:11.494 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:19:11.495 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-28T12:19:11.495 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:19:11.567 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rados 2024-03-28T12:19:12.128 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:14 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:19:12.224 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rados-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-28T12:19:12.277 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:19:12.278 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-28T12:19:12.278 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:19:12.353 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rgw 2024-03-28T12:19:12.916 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:14 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:19:13.013 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-28T12:19:13.065 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:19:13.066 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-28T12:19:13.066 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:19:13.137 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-cephfs 2024-03-28T12:19:13.701 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:15 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:19:13.797 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-cephfs-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-28T12:19:13.850 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:19:13.851 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-28T12:19:13.851 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:19:13.920 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rbd 2024-03-28T12:19:14.483 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:16 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:19:14.581 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-28T12:19:14.633 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:19:14.634 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-28T12:19:14.634 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:19:14.701 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-fuse 2024-03-28T12:19:15.260 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:17 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:19:15.408 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:19:15.408 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:19:15.408 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-03-28T12:19:15.408 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:19:15.408 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-28T12:19:15.409 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 90 k 2024-03-28T12:19:15.409 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:15.409 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-28T12:19:15.409 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:19:15.409 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-28T12:19:15.409 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:15.410 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 90 k 2024-03-28T12:19:15.410 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 313 k 2024-03-28T12:19:15.410 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-28T12:19:15.614 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-fuse-19.0.0-2473.g785fc5e8.el9.x86_64.rpm 438 kB/s | 90 kB 00:00 2024-03-28T12:19:15.615 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-28T12:19:15.615 INFO:teuthology.orchestra.run.smithi183.stdout:Total 436 kB/s | 90 kB 00:00 2024-03-28T12:19:15.615 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-28T12:19:15.623 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-28T12:19:15.624 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-28T12:19:15.681 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-28T12:19:15.681 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-28T12:19:15.828 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-28T12:19:15.938 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-28T12:19:16.347 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-28T12:19:16.653 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-28T12:19:16.653 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:16.653 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-28T12:19:16.654 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:19:16.654 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:16.654 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:19:16.909 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-mirror 2024-03-28T12:19:17.481 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:19 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:19:17.635 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:19:17.636 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:19:17.636 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-28T12:19:17.636 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:19:17.636 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-28T12:19:17.636 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 3.4 M 2024-03-28T12:19:17.636 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:17.636 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-28T12:19:17.636 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:19:17.636 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-28T12:19:17.636 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:17.637 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 3.4 M 2024-03-28T12:19:17.637 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 29 M 2024-03-28T12:19:17.637 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-28T12:19:18.042 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-mirror-19.0.0-2473.g785fc5e8.el9.x86_64.rpm 8.4 MB/s | 3.4 MB 00:00 2024-03-28T12:19:18.043 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-28T12:19:18.043 INFO:teuthology.orchestra.run.smithi183.stdout:Total 8.4 MB/s | 3.4 MB 00:00 2024-03-28T12:19:18.043 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-28T12:19:18.054 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-28T12:19:18.054 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-28T12:19:18.129 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-28T12:19:18.130 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-28T12:19:18.695 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-28T12:19:18.768 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-mirror-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-28T12:19:18.793 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-28T12:19:18.793 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-28T12:19:18.793 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-28T12:19:18.794 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T12:19:18.794 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-28T12:19:18.794 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:20.062 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-mirror-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-28T12:19:20.062 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:20.062 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-28T12:19:20.063 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:19:20.063 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:20.063 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:19:20.309 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-nbd 2024-03-28T12:19:20.891 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:22 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:19:21.042 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:19:21.043 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:19:21.043 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-03-28T12:19:21.043 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:19:21.043 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-28T12:19:21.043 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 176 k 2024-03-28T12:19:21.043 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:21.044 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-28T12:19:21.044 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:19:21.044 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-28T12:19:21.044 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:21.044 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 176 k 2024-03-28T12:19:21.044 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 673 k 2024-03-28T12:19:21.044 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-28T12:19:21.271 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-nbd-19.0.0-2473.g785fc5e8.el9.x86_64.rpm 771 kB/s | 176 kB 00:00 2024-03-28T12:19:21.272 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-28T12:19:21.272 INFO:teuthology.orchestra.run.smithi183.stdout:Total 766 kB/s | 176 kB 00:00 2024-03-28T12:19:21.272 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-28T12:19:21.282 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-28T12:19:21.282 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-28T12:19:21.338 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-28T12:19:21.338 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-28T12:19:21.487 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-28T12:19:21.596 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-nbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-28T12:19:22.046 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-28T12:19:22.325 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-nbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-28T12:19:22.326 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:22.326 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-28T12:19:22.326 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:19:22.326 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:22.326 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:19:22.567 DEBUG:teuthology.parallel:result is None 2024-03-28T12:19:22.567 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=785fc5e8d093374f751c7da3e51e9b56b3718d17 2024-03-28T12:19:22.784 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-28T12:19:22.804 INFO:teuthology.orchestra.run.smithi183.stdout:19.0.0-2473.g785fc5e8.el9 2024-03-28T12:19:22.805 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2473.g785fc5e8.el9 2024-03-28T12:19:22.805 INFO:teuthology.task.install:The correct ceph version 19.0.0-2473.g785fc5e8 is installed. 2024-03-28T12:19:22.807 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-28T12:19:22.807 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T12:19:22.807 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-28T12:19:22.872 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-28T12:19:22.872 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T12:19:22.872 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/daemon-helper 2024-03-28T12:19:22.939 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-28T12:19:23.005 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-28T12:19:23.005 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T12:19:23.005 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-28T12:19:23.071 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-28T12:19:23.138 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-28T12:19:23.138 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T12:19:23.138 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/stdin-killer 2024-03-28T12:19:23.205 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-28T12:19:23.273 INFO:teuthology.run_tasks:Running task ceph... 2024-03-28T12:19:23.358 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-28T12:19:23.358 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 777 /var/log/ceph 2024-03-28T12:19:23.385 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-28T12:19:23.385 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-28T12:19:23.452 INFO:tasks.ceph:Creating extra log directories... 2024-03-28T12:19:23.452 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-28T12:19:23.526 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-28T12:19:23.526 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-28T12:19:23.526 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627765', 'branch': 'ci-yingxin-crimson-load-obc-3', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write}', 'email': 'yingxin.cheng@intel.com', 'first_in_suite': False, 'job_id': '7627765', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'ci-yingxin-crimson-load-obc-3'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'ci-yingxin-crimson-load-obc-3', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'}}, 'owner': 'scheduled_yingxin@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'ci-yingxin-crimson-load-obc-3', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17', 'targets': {'smithi183.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGyf8yHXV/NPajLaUjsXrgBQJSrC9HHbq5O13zPEu3G6a09Xdx4y7dcnGZRoAn+B4cZgxQIsBsaXlkdJ6/oyt88='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-28_05:47:02', 'tube': 'smithi', 'user': 'yingxin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-28T12:19:23.527 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-28T12:19:23.574 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m0777 -- /var/run/ceph 2024-03-28T12:19:23.641 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T12:19:23.641 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/scratch_devs of=/dev/stdout 2024-03-28T12:19:23.696 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-28T12:19:23.696 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_1 2024-03-28T12:19:23.750 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-28T12:19:23.751 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T12:19:23.751 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-03-28T12:19:23.751 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T12:19:23.751 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T12:19:23.751 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-28 12:19:19.048996126 +0000 2024-03-28T12:19:23.751 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-28 12:16:16.665140708 +0000 2024-03-28T12:19:23.751 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-28 12:16:16.665140708 +0000 2024-03-28T12:19:23.751 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-28 12:16:16.665140708 +0000 2024-03-28T12:19:23.751 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-28T12:19:23.816 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-28T12:19:23.816 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-28T12:19:23.816 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000183251 s, 2.8 MB/s 2024-03-28T12:19:23.817 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-28T12:19:23.872 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_2 2024-03-28T12:19:23.928 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-28T12:19:23.928 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T12:19:23.928 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 681 Links: 1 2024-03-28T12:19:23.928 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T12:19:23.928 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T12:19:23.928 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-28 12:19:19.049996099 +0000 2024-03-28T12:19:23.928 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-28 12:16:17.012131231 +0000 2024-03-28T12:19:23.928 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-28 12:16:17.012131231 +0000 2024-03-28T12:19:23.928 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-28 12:16:17.012131231 +0000 2024-03-28T12:19:23.928 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-28T12:19:23.993 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-28T12:19:23.993 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-28T12:19:23.993 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000206926 s, 2.5 MB/s 2024-03-28T12:19:23.994 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-28T12:19:24.051 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_3 2024-03-28T12:19:24.105 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-28T12:19:24.105 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T12:19:24.106 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-03-28T12:19:24.106 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T12:19:24.106 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T12:19:24.106 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-28 12:19:19.049996099 +0000 2024-03-28T12:19:24.106 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-28 12:16:17.345122137 +0000 2024-03-28T12:19:24.106 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-28 12:16:17.345122137 +0000 2024-03-28T12:19:24.106 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-28 12:16:17.345122137 +0000 2024-03-28T12:19:24.106 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-28T12:19:24.171 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-28T12:19:24.171 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-28T12:19:24.171 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.00017748 s, 2.9 MB/s 2024-03-28T12:19:24.173 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-28T12:19:24.228 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_4 2024-03-28T12:19:24.282 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-28T12:19:24.283 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-28T12:19:24.283 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-03-28T12:19:24.283 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-28T12:19:24.283 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-28T12:19:24.283 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-28 12:19:19.049996099 +0000 2024-03-28T12:19:24.283 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-28 12:16:17.681112960 +0000 2024-03-28T12:19:24.283 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-28 12:16:17.681112960 +0000 2024-03-28T12:19:24.283 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-28 12:16:17.681112960 +0000 2024-03-28T12:19:24.283 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-28T12:19:24.348 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-28T12:19:24.348 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-28T12:19:24.348 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.00020845 s, 2.5 MB/s 2024-03-28T12:19:24.350 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-28T12:19:24.405 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-03-28T12:19:24.405 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi183.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-28T12:19:24.406 INFO:tasks.ceph:Generating config... 2024-03-28T12:19:24.407 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-28T12:19:24.407 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-28T12:19:24.407 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-28T12:19:24.407 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-28T12:19:24.407 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-28T12:19:24.407 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-03-28T12:19:24.407 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-03-28T12:19:24.407 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-03-28T12:19:24.407 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-28T12:19:24.407 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-28T12:19:24.407 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-03-28T12:19:24.407 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-03-28T12:19:24.408 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-03-28T12:19:24.408 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-03-28T12:19:24.408 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-28T12:19:24.408 INFO:tasks.ceph:[osd] err to stderr = True 2024-03-28T12:19:24.408 INFO:tasks.ceph:[osd] log flush on exit = True 2024-03-28T12:19:24.408 INFO:tasks.ceph:[osd] log to file = False 2024-03-28T12:19:24.408 INFO:tasks.ceph:[osd] log to stderr = True 2024-03-28T12:19:24.408 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-28T12:19:24.408 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-28T12:19:24.408 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-28T12:19:24.408 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-28T12:19:24.408 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-28T12:19:24.408 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-28T12:19:24.408 INFO:tasks.ceph:[global] auth client required = none 2024-03-28T12:19:24.409 INFO:tasks.ceph:[global] auth cluster required = none 2024-03-28T12:19:24.409 INFO:tasks.ceph:[global] auth service required = none 2024-03-28T12:19:24.409 INFO:tasks.ceph:[global] auth supported = none 2024-03-28T12:19:24.409 INFO:tasks.ceph:[global] debug asok = 0/0 2024-03-28T12:19:24.409 INFO:tasks.ceph:[global] debug auth = 0/0 2024-03-28T12:19:24.409 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-03-28T12:19:24.409 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-03-28T12:19:24.409 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-03-28T12:19:24.409 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-03-28T12:19:24.409 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-03-28T12:19:24.409 INFO:tasks.ceph:[global] debug client = 0/0 2024-03-28T12:19:24.409 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-03-28T12:19:24.409 INFO:tasks.ceph:[global] debug context = 0/0 2024-03-28T12:19:24.410 INFO:tasks.ceph:[global] debug crush = 0/0 2024-03-28T12:19:24.410 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-03-28T12:19:24.410 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-03-28T12:19:24.410 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-03-28T12:19:24.410 INFO:tasks.ceph:[global] debug filer = 0/0 2024-03-28T12:19:24.410 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-03-28T12:19:24.410 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-03-28T12:19:24.410 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-03-28T12:19:24.410 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-03-28T12:19:24.410 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-03-28T12:19:24.410 INFO:tasks.ceph:[global] debug journal = 0/0 2024-03-28T12:19:24.410 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-03-28T12:19:24.410 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-03-28T12:19:24.410 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-03-28T12:19:24.411 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-03-28T12:19:24.411 INFO:tasks.ceph:[global] debug mds = 0/0 2024-03-28T12:19:24.411 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-03-28T12:19:24.411 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-03-28T12:19:24.411 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-03-28T12:19:24.411 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-03-28T12:19:24.411 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-03-28T12:19:24.411 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-03-28T12:19:24.411 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-03-28T12:19:24.411 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-03-28T12:19:24.411 INFO:tasks.ceph:[global] debug mon = 0/0 2024-03-28T12:19:24.411 INFO:tasks.ceph:[global] debug monc = 0/0 2024-03-28T12:19:24.411 INFO:tasks.ceph:[global] debug ms = 0/0 2024-03-28T12:19:24.411 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-03-28T12:19:24.411 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-03-28T12:19:24.412 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-03-28T12:19:24.412 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-03-28T12:19:24.412 INFO:tasks.ceph:[global] debug osd = 0/0 2024-03-28T12:19:24.412 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-03-28T12:19:24.412 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-03-28T12:19:24.412 INFO:tasks.ceph:[global] debug rados = 0/0 2024-03-28T12:19:24.412 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-03-28T12:19:24.412 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-03-28T12:19:24.412 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-03-28T12:19:24.412 INFO:tasks.ceph:[global] debug refs = 0/0 2024-03-28T12:19:24.412 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-03-28T12:19:24.412 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2024-03-28T12:19:24.413 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2024-03-28T12:19:24.413 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-03-28T12:19:24.413 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-03-28T12:19:24.413 INFO:tasks.ceph:[global] debug striper = 0/0 2024-03-28T12:19:24.413 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-03-28T12:19:24.413 INFO:tasks.ceph:[global] debug timer = 0/0 2024-03-28T12:19:24.413 INFO:tasks.ceph:[global] debug tp = 0/0 2024-03-28T12:19:24.413 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-03-28T12:19:24.413 INFO:tasks.ceph:[global] ms client mode = crc 2024-03-28T12:19:24.413 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-03-28T12:19:24.413 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-03-28T12:19:24.413 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-03-28T12:19:24.413 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-03-28T12:19:24.413 INFO:tasks.ceph:[global] ms service mode = crc 2024-03-28T12:19:24.413 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-28T12:19:24.414 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-28T12:19:24.414 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-03-28T12:19:24.414 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-03-28T12:19:24.414 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-03-28T12:19:24.414 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-03-28T12:19:24.414 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-03-28T12:19:24.414 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-03-28T12:19:24.414 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-03-28T12:19:24.414 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-03-28T12:19:24.414 INFO:tasks.ceph:Setting up mon.a... 2024-03-28T12:19:24.414 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-28T12:19:24.511 INFO:teuthology.orchestra.run.smithi183.stdout:creating /etc/ceph/ceph.keyring 2024-03-28T12:19:24.517 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-28T12:19:24.570 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-28T12:19:24.596 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.183')] 2024-03-28T12:19:24.597 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.183', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-03-28T12:19:24.598 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T12:19:24.598 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-28T12:19:24.651 DEBUG:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.183 --print /home/ubuntu/cephtest/ceph.monmap 2024-03-28T12:19:24.742 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:24.742+0000 7fd6fc02af00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:24.742 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:24.743+0000 7fd6fc02af00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:24.742 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:24.743+0000 7fd6fc02af00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:24.743 INFO:teuthology.orchestra.run.smithi183.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-28T12:19:24.743 INFO:teuthology.orchestra.run.smithi183.stdout:monmaptool: generated fsid fb91711e-35b8-4926-b614-34a9862ce610 2024-03-28T12:19:24.743 INFO:teuthology.orchestra.run.smithi183.stdout:setting min_mon_release = quincy 2024-03-28T12:19:24.743 INFO:teuthology.orchestra.run.smithi183.stdout:epoch 0 2024-03-28T12:19:24.743 INFO:teuthology.orchestra.run.smithi183.stdout:fsid fb91711e-35b8-4926-b614-34a9862ce610 2024-03-28T12:19:24.743 INFO:teuthology.orchestra.run.smithi183.stdout:last_changed 2024-03-28T12:19:24.744187+0000 2024-03-28T12:19:24.744 INFO:teuthology.orchestra.run.smithi183.stdout:created 2024-03-28T12:19:24.744187+0000 2024-03-28T12:19:24.744 INFO:teuthology.orchestra.run.smithi183.stdout:min_mon_release 17 (quincy) 2024-03-28T12:19:24.744 INFO:teuthology.orchestra.run.smithi183.stdout:election_strategy: 1 2024-03-28T12:19:24.744 INFO:teuthology.orchestra.run.smithi183.stdout:0: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.a 2024-03-28T12:19:24.744 INFO:teuthology.orchestra.run.smithi183.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-03-28T12:19:24.745 DEBUG:teuthology.orchestra.run.smithi183:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-28T12:19:24.759 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID fb91711e-35b8-4926-b614-34a9862ce610... 2024-03-28T12:19:24.760 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-28T12:19:24.849 INFO:teuthology.orchestra.run.smithi183.stdout:[global] 2024-03-28T12:19:24.849 INFO:teuthology.orchestra.run.smithi183.stdout: chdir = "" 2024-03-28T12:19:24.849 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-28T12:19:24.849 INFO:teuthology.orchestra.run.smithi183.stdout: auth supported = none 2024-03-28T12:19:24.849 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.849 INFO:teuthology.orchestra.run.smithi183.stdout: filestore xattr use omap = true 2024-03-28T12:19:24.849 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.849 INFO:teuthology.orchestra.run.smithi183.stdout: mon clock drift allowed = 1.000 2024-03-28T12:19:24.849 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.849 INFO:teuthology.orchestra.run.smithi183.stdout: osd crush chooseleaf type = 0 2024-03-28T12:19:24.850 INFO:teuthology.orchestra.run.smithi183.stdout: auth debug = true 2024-03-28T12:19:24.850 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.850 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on old message = true 2024-03-28T12:19:24.850 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on bug = true 2024-03-28T12:19:24.850 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.850 INFO:teuthology.orchestra.run.smithi183.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-28T12:19:24.850 INFO:teuthology.orchestra.run.smithi183.stdout: mon pg warn max object skew = 0 2024-03-28T12:19:24.850 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.850 INFO:teuthology.orchestra.run.smithi183.stdout: # disable pg_autoscaler by default for new pools 2024-03-28T12:19:24.850 INFO:teuthology.orchestra.run.smithi183.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-28T12:19:24.850 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.850 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default size = 2 2024-03-28T12:19:24.850 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.850 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow primary affinity = true 2024-03-28T12:19:24.851 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow pg remap = true 2024-03-28T12:19:24.851 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on legacy crush tunables = false 2024-03-28T12:19:24.851 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on crush straw calc version zero = false 2024-03-28T12:19:24.851 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on no sortbitwise = false 2024-03-28T12:19:24.851 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on osd down out interval zero = false 2024-03-28T12:19:24.851 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on too few osds = false 2024-03-28T12:19:24.851 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-28T12:19:24.851 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-28T12:19:24.851 INFO:teuthology.orchestra.run.smithi183.stdout: mon_allow_pool_size_one = true 2024-03-28T12:19:24.851 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.851 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-28T12:19:24.851 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.851 INFO:teuthology.orchestra.run.smithi183.stdout: osd default data pool replay window = 5 2024-03-28T12:19:24.851 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.851 INFO:teuthology.orchestra.run.smithi183.stdout: mon allow pool delete = true 2024-03-28T12:19:24.852 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.852 INFO:teuthology.orchestra.run.smithi183.stdout: mon cluster log file level = debug 2024-03-28T12:19:24.852 INFO:teuthology.orchestra.run.smithi183.stdout: debug asserts on shutdown = true 2024-03-28T12:19:24.852 INFO:teuthology.orchestra.run.smithi183.stdout: mon health detail to clog = false 2024-03-28T12:19:24.852 INFO:teuthology.orchestra.run.smithi183.stdout: mon host = 172.21.15.183 2024-03-28T12:19:24.852 INFO:teuthology.orchestra.run.smithi183.stdout: auth client required = none 2024-03-28T12:19:24.852 INFO:teuthology.orchestra.run.smithi183.stdout: auth cluster required = none 2024-03-28T12:19:24.852 INFO:teuthology.orchestra.run.smithi183.stdout: auth service required = none 2024-03-28T12:19:24.852 INFO:teuthology.orchestra.run.smithi183.stdout: debug asok = 0/0 2024-03-28T12:19:24.852 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 0/0 2024-03-28T12:19:24.852 INFO:teuthology.orchestra.run.smithi183.stdout: debug bdev = 0/0 2024-03-28T12:19:24.852 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluefs = 0/0 2024-03-28T12:19:24.852 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluestore = 0/0 2024-03-28T12:19:24.852 INFO:teuthology.orchestra.run.smithi183.stdout: debug buffer = 0/0 2024-03-28T12:19:24.852 INFO:teuthology.orchestra.run.smithi183.stdout: debug civetweb = 0/0 2024-03-28T12:19:24.852 INFO:teuthology.orchestra.run.smithi183.stdout: debug client = 0/0 2024-03-28T12:19:24.852 INFO:teuthology.orchestra.run.smithi183.stdout: debug compressor = 0/0 2024-03-28T12:19:24.853 INFO:teuthology.orchestra.run.smithi183.stdout: debug context = 0/0 2024-03-28T12:19:24.853 INFO:teuthology.orchestra.run.smithi183.stdout: debug crush = 0/0 2024-03-28T12:19:24.853 INFO:teuthology.orchestra.run.smithi183.stdout: debug crypto = 0/0 2024-03-28T12:19:24.853 INFO:teuthology.orchestra.run.smithi183.stdout: debug dpdk = 0/0 2024-03-28T12:19:24.853 INFO:teuthology.orchestra.run.smithi183.stdout: debug eventtrace = 0/0 2024-03-28T12:19:24.853 INFO:teuthology.orchestra.run.smithi183.stdout: debug filer = 0/0 2024-03-28T12:19:24.853 INFO:teuthology.orchestra.run.smithi183.stdout: debug filestore = 0/0 2024-03-28T12:19:24.853 INFO:teuthology.orchestra.run.smithi183.stdout: debug finisher = 0/0 2024-03-28T12:19:24.853 INFO:teuthology.orchestra.run.smithi183.stdout: debug fuse = 0/0 2024-03-28T12:19:24.853 INFO:teuthology.orchestra.run.smithi183.stdout: debug heartbeatmap = 0/0 2024-03-28T12:19:24.853 INFO:teuthology.orchestra.run.smithi183.stdout: debug javaclient = 0/0 2024-03-28T12:19:24.853 INFO:teuthology.orchestra.run.smithi183.stdout: debug journal = 0/0 2024-03-28T12:19:24.853 INFO:teuthology.orchestra.run.smithi183.stdout: debug journaler = 0/0 2024-03-28T12:19:24.853 INFO:teuthology.orchestra.run.smithi183.stdout: debug kstore = 0/0 2024-03-28T12:19:24.853 INFO:teuthology.orchestra.run.smithi183.stdout: debug leveldb = 0/0 2024-03-28T12:19:24.853 INFO:teuthology.orchestra.run.smithi183.stdout: debug lockdep = 0/0 2024-03-28T12:19:24.854 INFO:teuthology.orchestra.run.smithi183.stdout: debug mds = 0/0 2024-03-28T12:19:24.854 INFO:teuthology.orchestra.run.smithi183.stdout: debug mds balancer = 0/0 2024-03-28T12:19:24.854 INFO:teuthology.orchestra.run.smithi183.stdout: debug mds locker = 0/0 2024-03-28T12:19:24.854 INFO:teuthology.orchestra.run.smithi183.stdout: debug mds log = 0/0 2024-03-28T12:19:24.854 INFO:teuthology.orchestra.run.smithi183.stdout: debug mds log expire = 0/0 2024-03-28T12:19:24.854 INFO:teuthology.orchestra.run.smithi183.stdout: debug mds migrator = 0/0 2024-03-28T12:19:24.854 INFO:teuthology.orchestra.run.smithi183.stdout: debug memdb = 0/0 2024-03-28T12:19:24.854 INFO:teuthology.orchestra.run.smithi183.stdout: debug mgr = 0/0 2024-03-28T12:19:24.854 INFO:teuthology.orchestra.run.smithi183.stdout: debug mgrc = 0/0 2024-03-28T12:19:24.854 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 0/0 2024-03-28T12:19:24.854 INFO:teuthology.orchestra.run.smithi183.stdout: debug monc = 0/0 2024-03-28T12:19:24.854 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 0/0 2024-03-28T12:19:24.854 INFO:teuthology.orchestra.run.smithi183.stdout: debug objclass = 0/0 2024-03-28T12:19:24.854 INFO:teuthology.orchestra.run.smithi183.stdout: debug objectcacher = 0/0 2024-03-28T12:19:24.854 INFO:teuthology.orchestra.run.smithi183.stdout: debug objecter = 0/0 2024-03-28T12:19:24.855 INFO:teuthology.orchestra.run.smithi183.stdout: debug optracker = 0/0 2024-03-28T12:19:24.855 INFO:teuthology.orchestra.run.smithi183.stdout: debug osd = 0/0 2024-03-28T12:19:24.855 INFO:teuthology.orchestra.run.smithi183.stdout: debug paxos = 0/0 2024-03-28T12:19:24.855 INFO:teuthology.orchestra.run.smithi183.stdout: debug perfcounter = 0/0 2024-03-28T12:19:24.855 INFO:teuthology.orchestra.run.smithi183.stdout: debug rados = 0/0 2024-03-28T12:19:24.855 INFO:teuthology.orchestra.run.smithi183.stdout: debug rbd = 0/0 2024-03-28T12:19:24.855 INFO:teuthology.orchestra.run.smithi183.stdout: debug rbd mirror = 0/0 2024-03-28T12:19:24.855 INFO:teuthology.orchestra.run.smithi183.stdout: debug rbd replay = 0/0 2024-03-28T12:19:24.855 INFO:teuthology.orchestra.run.smithi183.stdout: debug refs = 0/0 2024-03-28T12:19:24.855 INFO:teuthology.orchestra.run.smithi183.stdout: debug rgw = 0/0 2024-03-28T12:19:24.855 INFO:teuthology.orchestra.run.smithi183.stdout: debug rgw lifecycle = 0/0 2024-03-28T12:19:24.855 INFO:teuthology.orchestra.run.smithi183.stdout: debug rgw notification = 0/0 2024-03-28T12:19:24.855 INFO:teuthology.orchestra.run.smithi183.stdout: debug rgw sync = 0/0 2024-03-28T12:19:24.855 INFO:teuthology.orchestra.run.smithi183.stdout: debug rocksdb = 0/0 2024-03-28T12:19:24.855 INFO:teuthology.orchestra.run.smithi183.stdout: debug striper = 0/0 2024-03-28T12:19:24.855 INFO:teuthology.orchestra.run.smithi183.stdout: debug throttle = 0/0 2024-03-28T12:19:24.855 INFO:teuthology.orchestra.run.smithi183.stdout: debug timer = 0/0 2024-03-28T12:19:24.856 INFO:teuthology.orchestra.run.smithi183.stdout: debug tp = 0/0 2024-03-28T12:19:24.856 INFO:teuthology.orchestra.run.smithi183.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-03-28T12:19:24.856 INFO:teuthology.orchestra.run.smithi183.stdout: ms client mode = crc 2024-03-28T12:19:24.856 INFO:teuthology.orchestra.run.smithi183.stdout: ms cluster mode = crc 2024-03-28T12:19:24.856 INFO:teuthology.orchestra.run.smithi183.stdout: ms mon client mode = crc 2024-03-28T12:19:24.856 INFO:teuthology.orchestra.run.smithi183.stdout: ms mon cluster mode = crc 2024-03-28T12:19:24.856 INFO:teuthology.orchestra.run.smithi183.stdout: ms mon service mode = crc 2024-03-28T12:19:24.856 INFO:teuthology.orchestra.run.smithi183.stdout: ms service mode = crc 2024-03-28T12:19:24.856 INFO:teuthology.orchestra.run.smithi183.stdout: fsid = fb91711e-35b8-4926-b614-34a9862ce610 2024-03-28T12:19:24.856 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.856 INFO:teuthology.orchestra.run.smithi183.stdout:[osd] 2024-03-28T12:19:24.856 INFO:teuthology.orchestra.run.smithi183.stdout: osd journal size = 100 2024-03-28T12:19:24.856 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.856 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub load threshold = 5.0 2024-03-28T12:19:24.856 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub max interval = 600 2024-03-28T12:19:24.856 INFO:teuthology.orchestra.run.smithi183.stdout: osd mclock profile = high_recovery_ops 2024-03-28T12:19:24.856 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.857 INFO:teuthology.orchestra.run.smithi183.stdout: osd recover clone overlap = true 2024-03-28T12:19:24.857 INFO:teuthology.orchestra.run.smithi183.stdout: osd recovery max chunk = 1048576 2024-03-28T12:19:24.857 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.857 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug shutdown = true 2024-03-28T12:19:24.857 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug op order = true 2024-03-28T12:19:24.857 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify stray on activate = true 2024-03-28T12:19:24.857 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug trim objects = true 2024-03-28T12:19:24.857 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.857 INFO:teuthology.orchestra.run.smithi183.stdout: osd open classes on start = true 2024-03-28T12:19:24.857 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug pg log writeout = true 2024-03-28T12:19:24.857 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.857 INFO:teuthology.orchestra.run.smithi183.stdout: osd deep scrub update digest min age = 30 2024-03-28T12:19:24.857 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.857 INFO:teuthology.orchestra.run.smithi183.stdout: osd map max advance = 10 2024-03-28T12:19:24.857 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.857 INFO:teuthology.orchestra.run.smithi183.stdout: journal zero on create = true 2024-03-28T12:19:24.857 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.858 INFO:teuthology.orchestra.run.smithi183.stdout: filestore ondisk finisher threads = 3 2024-03-28T12:19:24.858 INFO:teuthology.orchestra.run.smithi183.stdout: filestore apply finisher threads = 3 2024-03-28T12:19:24.858 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.858 INFO:teuthology.orchestra.run.smithi183.stdout: bdev debug aio = true 2024-03-28T12:19:24.858 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug misdirected ops = true 2024-03-28T12:19:24.858 INFO:teuthology.orchestra.run.smithi183.stdout: debug monc = 20 2024-03-28T12:19:24.858 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore block size = 96636764160 2024-03-28T12:19:24.858 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore compression algorithm = snappy 2024-03-28T12:19:24.858 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore compression mode = aggressive 2024-03-28T12:19:24.858 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore fsck on mount = True 2024-03-28T12:19:24.858 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore rocksdb cf = False 2024-03-28T12:19:24.858 INFO:teuthology.orchestra.run.smithi183.stdout: crimson alien thread cpu cores = 6-7 2024-03-28T12:19:24.858 INFO:teuthology.orchestra.run.smithi183.stdout: crimson osd obc lru size = 10 2024-03-28T12:19:24.858 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluefs = 20 2024-03-28T12:19:24.858 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluestore = 20 2024-03-28T12:19:24.859 INFO:teuthology.orchestra.run.smithi183.stdout: debug filestore = 0/0 2024-03-28T12:19:24.859 INFO:teuthology.orchestra.run.smithi183.stdout: debug journal = 0/0 2024-03-28T12:19:24.859 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 0/0 2024-03-28T12:19:24.859 INFO:teuthology.orchestra.run.smithi183.stdout: debug osd = 0/0 2024-03-28T12:19:24.859 INFO:teuthology.orchestra.run.smithi183.stdout: debug rocksdb = 10 2024-03-28T12:19:24.859 INFO:teuthology.orchestra.run.smithi183.stdout: err to stderr = True 2024-03-28T12:19:24.859 INFO:teuthology.orchestra.run.smithi183.stdout: log flush on exit = True 2024-03-28T12:19:24.859 INFO:teuthology.orchestra.run.smithi183.stdout: log to file = False 2024-03-28T12:19:24.859 INFO:teuthology.orchestra.run.smithi183.stdout: log to stderr = True 2024-03-28T12:19:24.859 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd backfillfull_ratio = 0.85 2024-03-28T12:19:24.859 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd full ratio = 0.9 2024-03-28T12:19:24.859 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd nearfull ratio = 0.8 2024-03-28T12:19:24.859 INFO:teuthology.orchestra.run.smithi183.stdout: osd failsafe full ratio = 0.95 2024-03-28T12:19:24.859 INFO:teuthology.orchestra.run.smithi183.stdout: osd objectstore = bluestore 2024-03-28T12:19:24.859 INFO:teuthology.orchestra.run.smithi183.stdout: osd shutdown pgref assert = True 2024-03-28T12:19:24.859 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.859 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr] 2024-03-28T12:19:24.860 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-28T12:19:24.860 INFO:teuthology.orchestra.run.smithi183.stdout: debug mgr = 20 2024-03-28T12:19:24.860 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-28T12:19:24.860 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-28T12:19:24.860 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-28T12:19:24.860 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-28T12:19:24.860 INFO:teuthology.orchestra.run.smithi183.stdout: mgr/telemetry/nag = false 2024-03-28T12:19:24.860 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.860 INFO:teuthology.orchestra.run.smithi183.stdout:[mon] 2024-03-28T12:19:24.860 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 0/0 2024-03-28T12:19:24.860 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 0/0 2024-03-28T12:19:24.860 INFO:teuthology.orchestra.run.smithi183.stdout: debug paxos = 0/0 2024-03-28T12:19:24.860 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-28T12:19:24.860 INFO:teuthology.orchestra.run.smithi183.stdout: mon data avail warn = 5 2024-03-28T12:19:24.860 INFO:teuthology.orchestra.run.smithi183.stdout: mon mgr mkfs grace = 240 2024-03-28T12:19:24.860 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-28T12:19:24.860 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd reporter subtree level = osd 2024-03-28T12:19:24.861 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd prime pg temp = true 2024-03-28T12:19:24.861 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-28T12:19:24.861 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.861 INFO:teuthology.orchestra.run.smithi183.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-28T12:19:24.861 INFO:teuthology.orchestra.run.smithi183.stdout: auth mon ticket ttl = 660 # 11m 2024-03-28T12:19:24.861 INFO:teuthology.orchestra.run.smithi183.stdout: auth service ticket ttl = 240 # 4m 2024-03-28T12:19:24.861 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.861 INFO:teuthology.orchestra.run.smithi183.stdout: # don't complain about insecure global_id in the test suite 2024-03-28T12:19:24.861 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-28T12:19:24.861 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-28T12:19:24.861 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.861 INFO:teuthology.orchestra.run.smithi183.stdout: # 1m isn't quite enough 2024-03-28T12:19:24.861 INFO:teuthology.orchestra.run.smithi183.stdout: mon_down_mkfs_grace = 2m 2024-03-28T12:19:24.861 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.861 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_filestore_osds = false 2024-03-28T12:19:24.861 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default crimson = True 2024-03-28T12:19:24.862 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:24.862 INFO:teuthology.orchestra.run.smithi183.stdout:[client] 2024-03-28T12:19:24.862 INFO:teuthology.orchestra.run.smithi183.stdout: rgw cache enabled = true 2024-03-28T12:19:24.862 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable ops log = true 2024-03-28T12:19:24.862 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable usage log = true 2024-03-28T12:19:24.862 INFO:teuthology.orchestra.run.smithi183.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-28T12:19:24.862 INFO:teuthology.orchestra.run.smithi183.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-28T12:19:24.862 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.a] 2024-03-28T12:19:24.862 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.0] 2024-03-28T12:19:24.862 INFO:teuthology.orchestra.run.smithi183.stdout: crimson seastar cpu cores = 0-2 2024-03-28T12:19:24.862 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.1] 2024-03-28T12:19:24.862 INFO:teuthology.orchestra.run.smithi183.stdout: crimson seastar cpu cores = 3-5 2024-03-28T12:19:24.863 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.2] 2024-03-28T12:19:24.863 INFO:teuthology.orchestra.run.smithi183.stdout: crimson seastar cpu cores = 0-2 2024-03-28T12:19:24.863 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.3] 2024-03-28T12:19:24.863 INFO:teuthology.orchestra.run.smithi183.stdout: crimson seastar cpu cores = 3-5 2024-03-28T12:19:24.863 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-28T12:19:24.864 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-28T12:19:24.929 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-28T12:19:24.929 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T12:19:24.929 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-28T12:19:24.944 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T12:19:24.944 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-28T12:19:24.999 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:19:24.999 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T12:19:24.999 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-28T12:19:24.999 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-28T12:19:25.078 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T12:19:25.078 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-28T12:19:25.134 INFO:tasks.ceph:Setting up mon nodes... 2024-03-28T12:19:25.134 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-28T12:19:25.134 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-28T12:19:25.237 INFO:teuthology.orchestra.run.smithi183.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-28T12:19:25.240 INFO:tasks.ceph:Setting up mds nodes... 2024-03-28T12:19:25.240 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-28T12:19:25.240 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-28T12:19:25.290 INFO:teuthology.orchestra.run.smithi183.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-28T12:19:25.306 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-28T12:19:25.306 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi183.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-28T12:19:25.306 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-28T12:19:25.333 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-03-28T12:19:25.333 INFO:tasks.ceph:role: osd.0 2024-03-28T12:19:25.333 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:19:25.333 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-28T12:19:25.407 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T12:19:25.407 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T12:19:25.407 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T12:19:25.407 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T12:19:25.407 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T12:19:25.408 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-28T12:19:25.408 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T12:19:25.408 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T12:19:25.408 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T12:19:25.408 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T12:19:25.905 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-28T12:19:25.907 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-28T12:19:25.908 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-28T12:19:26.021 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-28T12:19:26.056 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-28T12:19:26.123 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-03-28T12:19:26.123 INFO:tasks.ceph:role: osd.1 2024-03-28T12:19:26.123 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:19:26.124 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-28T12:19:26.199 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T12:19:26.199 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T12:19:26.199 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T12:19:26.199 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T12:19:26.199 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T12:19:26.199 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-28T12:19:26.199 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T12:19:26.199 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T12:19:26.200 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T12:19:26.200 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T12:19:26.662 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-28T12:19:26.664 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-28T12:19:26.664 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-28T12:19:26.755 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-28T12:19:26.787 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-28T12:19:26.856 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-03-28T12:19:26.856 INFO:tasks.ceph:role: osd.2 2024-03-28T12:19:26.856 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:19:26.856 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-28T12:19:26.930 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-28T12:19:26.931 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-28T12:19:26.931 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-28T12:19:26.931 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-28T12:19:26.931 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-28T12:19:26.931 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-28T12:19:26.931 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-28T12:19:26.931 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-28T12:19:26.931 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-28T12:19:26.931 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-28T12:19:27.420 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-28T12:19:27.422 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-28T12:19:27.422 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-28T12:19:27.577 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-28T12:19:27.610 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T12:19:27.941 INFO:teuthology.orchestra.run.smithi183.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-28T12:19:27.942 INFO:teuthology.orchestra.run.smithi183.stderr:INFO 2024-03-28 12:19:27,943 seastar - Reactor backend: linux-aio 2024-03-28T12:19:27.960 INFO:teuthology.orchestra.run.smithi183.stderr:==29110==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-28T12:19:28.103 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG 2024-03-28 12:19:28,104 [shard 0:main] seastar - reactor::drain 2024-03-28T12:19:28.675 INFO:teuthology.orchestra.run.smithi183.stderr:----------------------------------------------------- 2024-03-28T12:19:28.675 INFO:teuthology.orchestra.run.smithi183.stderr:Suppressions used: 2024-03-28T12:19:28.675 INFO:teuthology.orchestra.run.smithi183.stderr: count bytes template 2024-03-28T12:19:28.675 INFO:teuthology.orchestra.run.smithi183.stderr: 1 8 MallocExtension::Register 2024-03-28T12:19:28.676 INFO:teuthology.orchestra.run.smithi183.stderr:----------------------------------------------------- 2024-03-28T12:19:28.676 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-28T12:19:28.729 INFO:teuthology.orchestra.run.smithi183.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-28T12:19:28.732 INFO:teuthology.orchestra.run.smithi183.stderr:INFO 2024-03-28 12:19:28,734 seastar - Reactor backend: linux-aio 2024-03-28T12:19:28.741 INFO:teuthology.orchestra.run.smithi183.stderr:==29108==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-28T12:19:33.388 INFO:teuthology.orchestra.run.smithi183.stderr:----------------------------------------------------- 2024-03-28T12:19:33.388 INFO:teuthology.orchestra.run.smithi183.stderr:Suppressions used: 2024-03-28T12:19:33.388 INFO:teuthology.orchestra.run.smithi183.stderr: count bytes template 2024-03-28T12:19:33.388 INFO:teuthology.orchestra.run.smithi183.stderr: 1 8 MallocExtension::Register 2024-03-28T12:19:33.388 INFO:teuthology.orchestra.run.smithi183.stderr:----------------------------------------------------- 2024-03-28T12:19:33.388 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-28T12:19:33.388 INFO:teuthology.orchestra.run.smithi183.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid fb91711e-35b8-4926-b614-34a9862ce610 2024-03-28T12:19:33.412 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-28T12:19:33.438 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T12:19:33.725 INFO:teuthology.orchestra.run.smithi183.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-28T12:19:33.726 INFO:teuthology.orchestra.run.smithi183.stderr:INFO 2024-03-28 12:19:33,727 seastar - Reactor backend: linux-aio 2024-03-28T12:19:33.731 INFO:teuthology.orchestra.run.smithi183.stderr:==29292==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-28T12:19:33.863 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG 2024-03-28 12:19:33,865 [shard 0:main] seastar - reactor::drain 2024-03-28T12:19:34.392 INFO:teuthology.orchestra.run.smithi183.stderr:----------------------------------------------------- 2024-03-28T12:19:34.392 INFO:teuthology.orchestra.run.smithi183.stderr:Suppressions used: 2024-03-28T12:19:34.392 INFO:teuthology.orchestra.run.smithi183.stderr: count bytes template 2024-03-28T12:19:34.392 INFO:teuthology.orchestra.run.smithi183.stderr: 1 8 MallocExtension::Register 2024-03-28T12:19:34.392 INFO:teuthology.orchestra.run.smithi183.stderr:----------------------------------------------------- 2024-03-28T12:19:34.392 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-28T12:19:34.447 INFO:teuthology.orchestra.run.smithi183.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-28T12:19:34.450 INFO:teuthology.orchestra.run.smithi183.stderr:INFO 2024-03-28 12:19:34,452 seastar - Reactor backend: linux-aio 2024-03-28T12:19:34.466 INFO:teuthology.orchestra.run.smithi183.stderr:==29290==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-28T12:19:39.113 INFO:teuthology.orchestra.run.smithi183.stderr:----------------------------------------------------- 2024-03-28T12:19:39.114 INFO:teuthology.orchestra.run.smithi183.stderr:Suppressions used: 2024-03-28T12:19:39.114 INFO:teuthology.orchestra.run.smithi183.stderr: count bytes template 2024-03-28T12:19:39.114 INFO:teuthology.orchestra.run.smithi183.stderr: 1 8 MallocExtension::Register 2024-03-28T12:19:39.114 INFO:teuthology.orchestra.run.smithi183.stderr:----------------------------------------------------- 2024-03-28T12:19:39.114 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-28T12:19:39.114 INFO:teuthology.orchestra.run.smithi183.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid fb91711e-35b8-4926-b614-34a9862ce610 2024-03-28T12:19:39.136 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-28T12:19:39.163 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-28T12:19:39.448 INFO:teuthology.orchestra.run.smithi183.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-28T12:19:39.449 INFO:teuthology.orchestra.run.smithi183.stderr:INFO 2024-03-28 12:19:39,451 seastar - Reactor backend: linux-aio 2024-03-28T12:19:39.454 INFO:teuthology.orchestra.run.smithi183.stderr:==29474==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-28T12:19:39.560 INFO:teuthology.orchestra.run.smithi183.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb975249 0xb847d10 0xb84928c 0xb8494a2 0xb8495f8 0xb849ac1 0x54daf 0xe194f 0x2a573 0xe0e5c 0x346a0c2 0x346a4be 0x346a5d7 0x5d5a784 0x5b9ee91 0x5bb9f55 0x6e94504 0x6ea5cd7 0x6ea7174 0x6ea71f9 0x6ea72fb 0x6ea73f3 0x6ea7600 0x6ea77e6 0x6ea7b35 0x6ea7ed9 0x6ea8109 0x6ea8373 0x6ea8640 0x6ea885e 0x6ea8916 0x6ea89dc 0x6ea8abd 0x5a05498 0x5a055a2 0x5a0565e 0x5a0571a 0x5a057d6 0x5a05923 0x5a05aae 0x5a05e6f 0x5a228ec 0xb85a995 0xb874c2f 0xb915c2d 0xb91727b 0xb5f39e3 0xb5f435f 0x3785266 0x3785dbf 0x3625d7a 0x3feaf 0x3ff5f 0x3412444 2024-03-28T12:19:39.560 INFO:teuthology.orchestra.run.smithi183.stderr:kernel callstack: 2024-03-28T12:19:39.588 INFO:teuthology.orchestra.run.smithi183.stderr:DEBUG 2024-03-28 12:19:39,589 [shard 0:main] seastar - reactor::drain 2024-03-28T12:19:40.123 INFO:teuthology.orchestra.run.smithi183.stderr:----------------------------------------------------- 2024-03-28T12:19:40.123 INFO:teuthology.orchestra.run.smithi183.stderr:Suppressions used: 2024-03-28T12:19:40.123 INFO:teuthology.orchestra.run.smithi183.stderr: count bytes template 2024-03-28T12:19:40.123 INFO:teuthology.orchestra.run.smithi183.stderr: 1 8 MallocExtension::Register 2024-03-28T12:19:40.123 INFO:teuthology.orchestra.run.smithi183.stderr:----------------------------------------------------- 2024-03-28T12:19:40.123 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-28T12:19:40.181 INFO:teuthology.orchestra.run.smithi183.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-28T12:19:40.184 INFO:teuthology.orchestra.run.smithi183.stderr:INFO 2024-03-28 12:19:40,186 seastar - Reactor backend: linux-aio 2024-03-28T12:19:40.199 INFO:teuthology.orchestra.run.smithi183.stderr:==29472==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-28T12:19:44.824 INFO:teuthology.orchestra.run.smithi183.stderr:----------------------------------------------------- 2024-03-28T12:19:44.824 INFO:teuthology.orchestra.run.smithi183.stderr:Suppressions used: 2024-03-28T12:19:44.824 INFO:teuthology.orchestra.run.smithi183.stderr: count bytes template 2024-03-28T12:19:44.824 INFO:teuthology.orchestra.run.smithi183.stderr: 1 8 MallocExtension::Register 2024-03-28T12:19:44.824 INFO:teuthology.orchestra.run.smithi183.stderr:----------------------------------------------------- 2024-03-28T12:19:44.824 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-28T12:19:44.825 INFO:teuthology.orchestra.run.smithi183.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid fb91711e-35b8-4926-b614-34a9862ce610 2024-03-28T12:19:44.847 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-28T12:19:44.873 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-28T12:19:44.873 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T12:19:44.873 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-28T12:19:44.940 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T12:19:44.940 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-28T12:19:45.006 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T12:19:45.006 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-28T12:19:45.072 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T12:19:45.072 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-28T12:19:45.138 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T12:19:45.138 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-28T12:19:45.193 INFO:tasks.ceph:Adding keys to all mons... 2024-03-28T12:19:45.193 DEBUG:teuthology.orchestra.run.smithi183:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-28T12:19:45.258 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr.x] 2024-03-28T12:19:45.258 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBNYAVmnb08DhAAEeASVDPVddMnMgKFDsndpA== 2024-03-28T12:19:45.258 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.0] 2024-03-28T12:19:45.258 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBQYAVmurg5NRAAlJaEZ7Zau2c0VRS67Ae/ow== 2024-03-28T12:19:45.258 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.1] 2024-03-28T12:19:45.258 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBWYAVmug65JBAABImWd+/jMvn94L8sw01z4g== 2024-03-28T12:19:45.258 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.2] 2024-03-28T12:19:45.258 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBcYAVmgKCSFBAAltXcjZTbVyRmRwZjzoJS+g== 2024-03-28T12:19:45.258 INFO:teuthology.orchestra.run.smithi183.stdout:[client.0] 2024-03-28T12:19:45.258 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBNYAVmTHFqERAApjvyWndlYs2ZtFCUMcWkfw== 2024-03-28T12:19:45.259 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-28T12:19:45.361 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T12:19:45.422 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T12:19:45.483 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-28T12:19:45.544 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-28T12:19:45.608 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-28T12:19:45.609 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-28T12:19:45.635 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-28T12:19:45.748 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:45.749+0000 7f33c3c36a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:45.748 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:45.749+0000 7f33c3c36a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:45.762 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-28T12:19:45.789 DEBUG:teuthology.orchestra.run.smithi183:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-28T12:19:45.843 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-28T12:19:45.844 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-28T12:19:45.844 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-28T12:19:45.886 INFO:tasks.ceph.mon.a:Started 2024-03-28T12:19:45.886 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-28T12:19:45.886 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-28T12:19:45.886 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-28T12:19:45.888 INFO:tasks.ceph.mgr.x:Started 2024-03-28T12:19:45.889 INFO:tasks.ceph:Setting crush tunables to default 2024-03-28T12:19:45.889 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd crush tunables default 2024-03-28T12:19:45.971 INFO:tasks.ceph.mon.a.smithi183.stderr:2024-03-28T12:19:45.972+0000 7f29a4a6ea80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:45.971 INFO:tasks.ceph.mon.a.smithi183.stderr:2024-03-28T12:19:45.972+0000 7f29a4a6ea80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:45.972 INFO:tasks.ceph.mon.a.smithi183.stderr:2024-03-28T12:19:45.972+0000 7f29a4a6ea80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:45.972 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:45.973+0000 7f40aabc7000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:46.001 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:45.997+0000 7ff11bd5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:46.015 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:46.016+0000 7ff11bd5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:46.083 INFO:teuthology.orchestra.run.smithi183.stderr:adjusted tunables profile to default 2024-03-28T12:19:46.094 INFO:tasks.ceph:check_enable_crimson: True 2024-03-28T12:19:46.095 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-03-28T12:19:46.095 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-03-28T12:19:46.176 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:46.176+0000 7f40aabc7000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:46.176 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:46.177+0000 7f40aabc7000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:46.204 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:46.200+0000 7f24020ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:46.206 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:46.207+0000 7f24020ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:46.304 INFO:tasks.ceph.mon.a.smithi183.stderr:2024-03-28T12:19:46.304+0000 7f299b7f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-03-28T12:19:46.304 INFO:tasks.ceph.mon.a.smithi183.stderr:Please be aware that this feature is experimental, untested, 2024-03-28T12:19:46.304 INFO:tasks.ceph.mon.a.smithi183.stderr:unsupported, and may result in data corruption, data loss, 2024-03-28T12:19:46.304 INFO:tasks.ceph.mon.a.smithi183.stderr:and/or irreparable damage to your cluster. Do not use 2024-03-28T12:19:46.304 INFO:tasks.ceph.mon.a.smithi183.stderr:feature with important data. 2024-03-28T12:19:46.304 INFO:tasks.ceph.mon.a.smithi183.stderr: 2024-03-28T12:19:46.664 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:46.664+0000 7f40aabc7000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-28T12:19:46.718 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:46.719+0000 7f40aabc7000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-28T12:19:47.095 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-28T12:19:47.096 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T12:19:47.096 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-28T12:19:47.123 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T12:19:47.123 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-28T12:19:47.189 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T12:19:47.190 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-28T12:19:47.253 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:47.254+0000 7f40aabc7000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-28T12:19:47.257 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 73dc0235-2068-46d3-96cc-f650a3358b76 0 2024-03-28T12:19:47.309 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:47.310+0000 7f40aabc7000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-28T12:19:47.357 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:47.357+0000 7f40aabc7000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-28T12:19:47.408 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:47.403+0000 7fdb61024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:47.410 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:47.410+0000 7fdb61024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:47.429 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:47.430+0000 7f40aabc7000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-28T12:19:47.474 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:47.474+0000 7f40aabc7000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-28T12:19:47.488 INFO:teuthology.orchestra.run.smithi183.stdout:0 2024-03-28T12:19:47.499 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 17f6fb8c-c31e-43a1-a1c5-b3102b59883d 1 2024-03-28T12:19:47.596 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:47.596+0000 7f40aabc7000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-28T12:19:47.649 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:47.644+0000 7fe265e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:47.651 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:47.652+0000 7fe265e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:47.671 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:47.672+0000 7f40aabc7000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-28T12:19:47.719 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:47.719+0000 7f40aabc7000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-28T12:19:47.729 INFO:teuthology.orchestra.run.smithi183.stdout:1 2024-03-28T12:19:47.741 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 6e7cc685-cb3b-41c2-85fa-f8027c3d26fd 2 2024-03-28T12:19:47.775 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:47.776+0000 7f40aabc7000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-28T12:19:47.826 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:47.827+0000 7f40aabc7000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-28T12:19:47.851 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:47.846+0000 7f186f31d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:47.853 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:47.854+0000 7f186f31d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:47.931 INFO:teuthology.orchestra.run.smithi183.stdout:2 2024-03-28T12:19:47.942 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-03-28T12:19:48.014 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-28T12:19:48.014 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-03-28T12:19:48.056 INFO:tasks.ceph.osd.0:Started 2024-03-28T12:19:48.056 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-03-28T12:19:48.089 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-28T12:19:48.089 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-03-28T12:19:48.131 INFO:tasks.ceph.osd.1:Started 2024-03-28T12:19:48.131 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-03-28T12:19:48.162 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-28T12:19:48.163 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-03-28T12:19:48.205 INFO:tasks.ceph.osd.2:Started 2024-03-28T12:19:48.206 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T12:19:48.243 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:48.244+0000 7f40aabc7000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-28T12:19:48.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:48.373+0000 7f8ae746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:48.380 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:48.381+0000 7f8ae746d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:48.435 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:48.436+0000 7f40aabc7000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-28T12:19:48.454 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:48.454 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":6,"fsid":"fb91711e-35b8-4926-b614-34a9862ce610","created":"2024-03-28T12:19:46.009730+0000","modified":"2024-03-28T12:19:47.932118+0000","last_up_change":"0.000000","last_in_change":"2024-03-28T12:19:47.932118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"73dc0235-2068-46d3-96cc-f650a3358b76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"17f6fb8c-c31e-43a1-a1c5-b3102b59883d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6e7cc685-cb3b-41c2-85fa-f8027c3d26fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T12:19:48.467 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-28T12:19:48.467 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-28T12:19:48.565 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:48.565+0000 7f40aabc7000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-28T12:19:48.613 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:48.614+0000 7f40aabc7000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-28T12:19:48.734 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:48.734+0000 7f40aabc7000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-28T12:19:48.769 DEBUG:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-28T12:19:48.839 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:48.840+0000 7f40aabc7000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-28T12:19:48.873 INFO:teuthology.misc.health.smithi183.stderr:2024-03-28T12:19:48.870+0000 7fbcb242d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:48.875 INFO:teuthology.misc.health.smithi183.stderr:2024-03-28T12:19:48.876+0000 7fbcb242d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:48.942 INFO:teuthology.misc.health.smithi183.stdout: 2024-03-28T12:19:48.942 INFO:teuthology.misc.health.smithi183.stdout:{"epoch":6,"fsid":"fb91711e-35b8-4926-b614-34a9862ce610","created":"2024-03-28T12:19:46.009730+0000","modified":"2024-03-28T12:19:47.932118+0000","last_up_change":"0.000000","last_in_change":"2024-03-28T12:19:47.932118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"73dc0235-2068-46d3-96cc-f650a3358b76","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"17f6fb8c-c31e-43a1-a1c5-b3102b59883d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6e7cc685-cb3b-41c2-85fa-f8027c3d26fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T12:19:48.953 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-03-28T12:19:49.010 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:49.011+0000 7f40aabc7000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-28T12:19:49.057 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:49.058+0000 7f40aabc7000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-28T12:19:49.228 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:49.229+0000 7f40aabc7000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-28T12:19:50.353 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:50.353+0000 7f40aabc7000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-28T12:19:50.668 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:50.669+0000 7f40aabc7000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-28T12:19:50.739 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:50.740+0000 7f40aabc7000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-28T12:19:50.867 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:50.868+0000 7f40aabc7000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-28T12:19:51.888 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-28T12:19:51.888+0000 7f4049d51640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-28T12:19:55.255 DEBUG:teuthology.orchestra.run.smithi183:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-28T12:19:55.363 INFO:teuthology.misc.health.smithi183.stderr:2024-03-28T12:19:55.361+0000 7f93be5f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:55.365 INFO:teuthology.misc.health.smithi183.stderr:2024-03-28T12:19:55.366+0000 7f93be5f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:55.551 INFO:teuthology.misc.health.smithi183.stdout: 2024-03-28T12:19:55.551 INFO:teuthology.misc.health.smithi183.stdout:{"epoch":8,"fsid":"fb91711e-35b8-4926-b614-34a9862ce610","created":"2024-03-28T12:19:46.009730+0000","modified":"2024-03-28T12:19:53.890047+0000","last_up_change":"2024-03-28T12:19:53.890047+0000","last_in_change":"2024-03-28T12:19:47.932118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"73dc0235-2068-46d3-96cc-f650a3358b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6803","nonce":3210039029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3210039029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6813","nonce":3210039029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":3210039029}]},"public_addr":"172.21.15.183:6803/3210039029","cluster_addr":"172.21.15.183:6802/3210039029","heartbeat_back_addr":"172.21.15.183:6813/3210039029","heartbeat_front_addr":"172.21.15.183:6812/3210039029","state":["exists","up"]},{"osd":1,"uuid":"17f6fb8c-c31e-43a1-a1c5-b3102b59883d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2887612789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6807","nonce":2887612789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2887612789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6811","nonce":2887612789}]},"public_addr":"172.21.15.183:6806/2887612789","cluster_addr":"172.21.15.183:6807/2887612789","heartbeat_back_addr":"172.21.15.183:6810/2887612789","heartbeat_front_addr":"172.21.15.183:6811/2887612789","state":["exists","up"]},{"osd":2,"uuid":"6e7cc685-cb3b-41c2-85fa-f8027c3d26fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2085197246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6805","nonce":2085197246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2085197246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6815","nonce":2085197246}]},"public_addr":"172.21.15.183:6804/2085197246","cluster_addr":"172.21.15.183:6805/2085197246","heartbeat_back_addr":"172.21.15.183:6816/2085197246","heartbeat_front_addr":"172.21.15.183:6815/2085197246","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T12:19:55.563 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-03-28T12:19:55.563 INFO:tasks.ceph:Creating RBD pool 2024-03-28T12:19:55.564 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd pool create rbd 8 2024-03-28T12:19:55.674 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:55.669+0000 7f1d6626d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:55.676 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:55.677+0000 7f1d6626d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:55.899 INFO:teuthology.orchestra.run.smithi183.stderr:pool 'rbd' created 2024-03-28T12:19:55.917 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-03-28T12:19:56.049 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:56.041+0000 7f60dee6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:56.051 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:56.052+0000 7f60dee6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:56.900 INFO:teuthology.orchestra.run.smithi183.stderr:enabled application 'rbd' on pool 'rbd' 2024-03-28T12:19:56.921 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-28T12:19:56.921 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-28T12:19:56.921 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-28T12:19:56.921 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-28T12:19:56.922 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-28T12:19:57.085 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:57.081+0000 7faa61f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:57.087 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:57.087+0000 7faa61f5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:57.385 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:57.403 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":4,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3156474871},{"type":"v1","addr":"172.21.15.183:6801","nonce":3156474871}]},"active_addr":"172.21.15.183:6801/3156474871","active_change":"2024-03-28T12:19:50.881814+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.183:0","nonce":2543380771}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.183:0","nonce":4189910744}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.183:0","nonce":1430338849}]}]} 2024-03-28T12:19:57.407 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-28T12:19:57.407 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-28T12:19:57.407 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T12:19:57.541 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:57.536+0000 7f8e68f14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:57.543 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:57.543+0000 7f8e68f14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:57.767 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:57.767 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":10,"fsid":"fb91711e-35b8-4926-b614-34a9862ce610","created":"2024-03-28T12:19:46.009730+0000","modified":"2024-03-28T12:19:56.900690+0000","last_up_change":"2024-03-28T12:19:53.890047+0000","last_in_change":"2024-03-28T12:19:47.932118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T12:19:54.911172+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-28T12:19:55.862989+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"73dc0235-2068-46d3-96cc-f650a3358b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6803","nonce":3210039029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3210039029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6813","nonce":3210039029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":3210039029}]},"public_addr":"172.21.15.183:6803/3210039029","cluster_addr":"172.21.15.183:6802/3210039029","heartbeat_back_addr":"172.21.15.183:6813/3210039029","heartbeat_front_addr":"172.21.15.183:6812/3210039029","state":["exists","up"]},{"osd":1,"uuid":"17f6fb8c-c31e-43a1-a1c5-b3102b59883d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2887612789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6807","nonce":2887612789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2887612789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6811","nonce":2887612789}]},"public_addr":"172.21.15.183:6806/2887612789","cluster_addr":"172.21.15.183:6807/2887612789","heartbeat_back_addr":"172.21.15.183:6810/2887612789","heartbeat_front_addr":"172.21.15.183:6811/2887612789","state":["exists","up"]},{"osd":2,"uuid":"6e7cc685-cb3b-41c2-85fa-f8027c3d26fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2085197246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6805","nonce":2085197246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2085197246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6815","nonce":2085197246}]},"public_addr":"172.21.15.183:6804/2085197246","cluster_addr":"172.21.15.183:6805/2085197246","heartbeat_back_addr":"172.21.15.183:6816/2085197246","heartbeat_front_addr":"172.21.15.183:6815/2085197246","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T12:19:57.783 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-28T12:19:57.783 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-28T12:19:57.906 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:57.902+0000 7f72e902d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:57.908 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:57.908+0000 7f72e902d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:58.097 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:58.097 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":11,"fsid":"fb91711e-35b8-4926-b614-34a9862ce610","created":"2024-03-28T12:19:46.009730+0000","modified":"2024-03-28T12:19:57.904450+0000","last_up_change":"2024-03-28T12:19:53.890047+0000","last_in_change":"2024-03-28T12:19:47.932118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-28T12:19:54.911172+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-28T12:19:55.862989+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"73dc0235-2068-46d3-96cc-f650a3358b76","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6803","nonce":3210039029}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":3210039029}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6813","nonce":3210039029}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":3210039029}]},"public_addr":"172.21.15.183:6803/3210039029","cluster_addr":"172.21.15.183:6802/3210039029","heartbeat_back_addr":"172.21.15.183:6813/3210039029","heartbeat_front_addr":"172.21.15.183:6812/3210039029","state":["exists","up"]},{"osd":1,"uuid":"17f6fb8c-c31e-43a1-a1c5-b3102b59883d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2887612789}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6807","nonce":2887612789}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":2887612789}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6811","nonce":2887612789}]},"public_addr":"172.21.15.183:6806/2887612789","cluster_addr":"172.21.15.183:6807/2887612789","heartbeat_back_addr":"172.21.15.183:6810/2887612789","heartbeat_front_addr":"172.21.15.183:6811/2887612789","state":["exists","up"]},{"osd":2,"uuid":"6e7cc685-cb3b-41c2-85fa-f8027c3d26fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2085197246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6805","nonce":2085197246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2085197246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6815","nonce":2085197246}]},"public_addr":"172.21.15.183:6804/2085197246","cluster_addr":"172.21.15.183:6805/2085197246","heartbeat_back_addr":"172.21.15.183:6816/2085197246","heartbeat_front_addr":"172.21.15.183:6815/2085197246","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-28T12:19:58.110 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-28T12:19:58.110 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-28T12:19:58.111 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-28T12:19:58.228 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:58.223+0000 7fb5dd1d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:58.228 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:58.223+0000 7f7a4f4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:58.228 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:58.223+0000 7f4091d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:58.230 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:58.230+0000 7f4091d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:58.230 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:58.230+0000 7fb5dd1d7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:58.231 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:58.230+0000 7f7a4f4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:58.258 INFO:teuthology.orchestra.run.smithi183.stdout:34359738372 2024-03-28T12:19:58.258 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-28T12:19:58.262 INFO:teuthology.orchestra.run.smithi183.stdout:34359738372 2024-03-28T12:19:58.263 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-28T12:19:58.265 INFO:teuthology.orchestra.run.smithi183.stdout:34359738372 2024-03-28T12:19:58.265 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-28T12:19:58.374 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:58.369+0000 7f12475f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:58.377 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:58.376+0000 7f12475f5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:58.420 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:58.416+0000 7f93bb26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:58.421 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:58.416+0000 7f6fa89e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:58.423 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:58.423+0000 7f6fa89e7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:58.424 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:58.423+0000 7f93bb26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:58.572 INFO:teuthology.orchestra.run.smithi183.stdout:34359738372 2024-03-28T12:19:58.585 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.1 2024-03-28T12:19:58.586 DEBUG:teuthology.parallel:result is None 2024-03-28T12:19:58.619 INFO:teuthology.orchestra.run.smithi183.stdout:34359738372 2024-03-28T12:19:58.623 INFO:teuthology.orchestra.run.smithi183.stdout:34359738372 2024-03-28T12:19:58.633 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.2 2024-03-28T12:19:58.634 DEBUG:teuthology.parallel:result is None 2024-03-28T12:19:58.637 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.0 2024-03-28T12:19:58.637 DEBUG:teuthology.parallel:result is None 2024-03-28T12:19:58.637 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-28T12:19:58.637 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T12:19:58.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:58.785+0000 7f67fca13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:58.790 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:58.790+0000 7f67fca13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:58.982 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:58.983 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-03-28T12:19:58.995 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-03-28T12:19:58.885788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80876,"kb_used_data":380,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034644,"statfs":{"total":289910292480,"available":289827475456,"internally_reserved":0,"allocated":389120,"data_stored":1240873,"data_compressed":59918,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.632158"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-28T12:19:57.919830+0000","last_change":"2024-03-28T12:19:57.043792+0000","last_active":"2024-03-28T12:19:57.919830+0000","last_peered":"2024-03-28T12:19:57.919830+0000","last_clean":"2024-03-28T12:19:57.919830+0000","last_became_active":"2024-03-28T12:19:57.041659+0000","last_became_peered":"2024-03-28T12:19:57.041659+0000","last_unstale":"2024-03-28T12:19:57.919830+0000","last_undegraded":"2024-03-28T12:19:57.919830+0000","last_fullsized":"2024-03-28T12:19:57.919830+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_clean_scrub_stamp":"2024-03-28T12:19:55.894779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-28T12:19:57.916739+0000","last_change":"2024-03-28T12:19:56.986440+0000","last_active":"2024-03-28T12:19:57.916739+0000","last_peered":"2024-03-28T12:19:57.916739+0000","last_clean":"2024-03-28T12:19:57.916739+0000","last_became_active":"2024-03-28T12:19:56.983271+0000","last_became_peered":"2024-03-28T12:19:56.983271+0000","last_unstale":"2024-03-28T12:19:57.916739+0000","last_undegraded":"2024-03-28T12:19:57.916739+0000","last_fullsized":"2024-03-28T12:19:57.916739+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_clean_scrub_stamp":"2024-03-28T12:19:55.894779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-28T12:19:57.918167+0000","last_change":"2024-03-28T12:19:56.999508+0000","last_active":"2024-03-28T12:19:57.918167+0000","last_peered":"2024-03-28T12:19:57.918167+0000","last_clean":"2024-03-28T12:19:57.918167+0000","last_became_active":"2024-03-28T12:19:56.996590+0000","last_became_peered":"2024-03-28T12:19:56.996590+0000","last_unstale":"2024-03-28T12:19:57.918167+0000","last_undegraded":"2024-03-28T12:19:57.918167+0000","last_fullsized":"2024-03-28T12:19:57.918167+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_clean_scrub_stamp":"2024-03-28T12:19:55.894779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-28T12:19:57.920849+0000","last_change":"2024-03-28T12:19:57.011721+0000","last_active":"2024-03-28T12:19:57.920849+0000","last_peered":"2024-03-28T12:19:57.920849+0000","last_clean":"2024-03-28T12:19:57.920849+0000","last_became_active":"2024-03-28T12:19:57.009407+0000","last_became_peered":"2024-03-28T12:19:57.009407+0000","last_unstale":"2024-03-28T12:19:57.920849+0000","last_undegraded":"2024-03-28T12:19:57.920849+0000","last_fullsized":"2024-03-28T12:19:57.920849+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_clean_scrub_stamp":"2024-03-28T12:19:55.894779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-03-28T12:19:57.012455+0000","last_change":"2024-03-28T12:19:57.013112+0000","last_active":"2024-03-28T12:19:57.012455+0000","last_peered":"2024-03-28T12:19:57.012455+0000","last_clean":"2024-03-28T12:19:55.894779+0000","last_became_active":"2024-03-28T12:19:57.007966+0000","last_became_peered":"2024-03-28T12:19:57.007966+0000","last_unstale":"2024-03-28T12:19:57.012455+0000","last_undegraded":"2024-03-28T12:19:57.012455+0000","last_fullsized":"2024-03-28T12:19:57.012455+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_clean_scrub_stamp":"2024-03-28T12:19:55.894779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-03-28T12:19:56.986516+0000","last_change":"2024-03-28T12:19:56.986661+0000","last_active":"2024-03-28T12:19:56.986516+0000","last_peered":"2024-03-28T12:19:56.986516+0000","last_clean":"2024-03-28T12:19:55.894779+0000","last_became_active":"2024-03-28T12:19:56.981775+0000","last_became_peered":"2024-03-28T12:19:56.981775+0000","last_unstale":"2024-03-28T12:19:56.986516+0000","last_undegraded":"2024-03-28T12:19:56.986516+0000","last_fullsized":"2024-03-28T12:19:56.986516+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_clean_scrub_stamp":"2024-03-28T12:19:55.894779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-03-28T12:19:56.980265+0000","last_change":"2024-03-28T12:19:56.980607+0000","last_active":"2024-03-28T12:19:56.980265+0000","last_peered":"2024-03-28T12:19:56.980265+0000","last_clean":"2024-03-28T12:19:55.894779+0000","last_became_active":"2024-03-28T12:19:56.975210+0000","last_became_peered":"2024-03-28T12:19:56.975210+0000","last_unstale":"2024-03-28T12:19:56.980265+0000","last_undegraded":"2024-03-28T12:19:56.980265+0000","last_fullsized":"2024-03-28T12:19:56.980265+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_clean_scrub_stamp":"2024-03-28T12:19:55.894779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-28T12:19:57.920420+0000","last_change":"2024-03-28T12:19:56.967505+0000","last_active":"2024-03-28T12:19:57.920420+0000","last_peered":"2024-03-28T12:19:57.920420+0000","last_clean":"2024-03-28T12:19:57.920420+0000","last_became_active":"2024-03-28T12:19:56.965117+0000","last_became_peered":"2024-03-28T12:19:56.965117+0000","last_unstale":"2024-03-28T12:19:57.920420+0000","last_undegraded":"2024-03-28T12:19:57.920420+0000","last_fullsized":"2024-03-28T12:19:57.920420+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_clean_scrub_stamp":"2024-03-28T12:19:55.894779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":138,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-28T12:19:57.917612+0000","last_change":"2024-03-28T12:19:57.027039+0000","last_active":"2024-03-28T12:19:57.917612+0000","last_peered":"2024-03-28T12:19:57.917612+0000","last_clean":"2024-03-28T12:19:57.917612+0000","last_became_active":"2024-03-28T12:19:57.022989+0000","last_became_peered":"2024-03-28T12:19:57.022989+0000","last_unstale":"2024-03-28T12:19:57.917612+0000","last_undegraded":"2024-03-28T12:19:57.917612+0000","last_fullsized":"2024-03-28T12:19:57.917612+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_clean_scrub_stamp":"2024-03-28T12:19:55.894779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26908,"kb_used_data":84,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344932,"statfs":{"total":96636764160,"available":96609210368,"internally_reserved":0,"allocated":86016,"data_stored":18958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26988,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344852,"statfs":{"total":96636764160,"available":96609128448,"internally_reserved":0,"allocated":155648,"data_stored":612589,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26980,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344860,"statfs":{"total":96636764160,"available":96609136640,"internally_reserved":0,"allocated":147456,"data_stored":609326,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-28T12:19:58.996 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-28T12:19:59.106 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:59.102+0000 7f698146d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:59.108 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:59.107+0000 7f698146d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:59.291 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:59.291 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-03-28T12:19:59.304 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-03-28T12:19:58.885788+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80876,"kb_used_data":380,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034644,"statfs":{"total":289910292480,"available":289827475456,"internally_reserved":0,"allocated":389120,"data_stored":1240873,"data_compressed":59918,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.632158"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-28T12:19:57.919830+0000","last_change":"2024-03-28T12:19:57.043792+0000","last_active":"2024-03-28T12:19:57.919830+0000","last_peered":"2024-03-28T12:19:57.919830+0000","last_clean":"2024-03-28T12:19:57.919830+0000","last_became_active":"2024-03-28T12:19:57.041659+0000","last_became_peered":"2024-03-28T12:19:57.041659+0000","last_unstale":"2024-03-28T12:19:57.919830+0000","last_undegraded":"2024-03-28T12:19:57.919830+0000","last_fullsized":"2024-03-28T12:19:57.919830+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_clean_scrub_stamp":"2024-03-28T12:19:55.894779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-28T12:19:57.916739+0000","last_change":"2024-03-28T12:19:56.986440+0000","last_active":"2024-03-28T12:19:57.916739+0000","last_peered":"2024-03-28T12:19:57.916739+0000","last_clean":"2024-03-28T12:19:57.916739+0000","last_became_active":"2024-03-28T12:19:56.983271+0000","last_became_peered":"2024-03-28T12:19:56.983271+0000","last_unstale":"2024-03-28T12:19:57.916739+0000","last_undegraded":"2024-03-28T12:19:57.916739+0000","last_fullsized":"2024-03-28T12:19:57.916739+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_clean_scrub_stamp":"2024-03-28T12:19:55.894779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-28T12:19:57.918167+0000","last_change":"2024-03-28T12:19:56.999508+0000","last_active":"2024-03-28T12:19:57.918167+0000","last_peered":"2024-03-28T12:19:57.918167+0000","last_clean":"2024-03-28T12:19:57.918167+0000","last_became_active":"2024-03-28T12:19:56.996590+0000","last_became_peered":"2024-03-28T12:19:56.996590+0000","last_unstale":"2024-03-28T12:19:57.918167+0000","last_undegraded":"2024-03-28T12:19:57.918167+0000","last_fullsized":"2024-03-28T12:19:57.918167+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_clean_scrub_stamp":"2024-03-28T12:19:55.894779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-28T12:19:57.920849+0000","last_change":"2024-03-28T12:19:57.011721+0000","last_active":"2024-03-28T12:19:57.920849+0000","last_peered":"2024-03-28T12:19:57.920849+0000","last_clean":"2024-03-28T12:19:57.920849+0000","last_became_active":"2024-03-28T12:19:57.009407+0000","last_became_peered":"2024-03-28T12:19:57.009407+0000","last_unstale":"2024-03-28T12:19:57.920849+0000","last_undegraded":"2024-03-28T12:19:57.920849+0000","last_fullsized":"2024-03-28T12:19:57.920849+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_clean_scrub_stamp":"2024-03-28T12:19:55.894779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-03-28T12:19:57.012455+0000","last_change":"2024-03-28T12:19:57.013112+0000","last_active":"2024-03-28T12:19:57.012455+0000","last_peered":"2024-03-28T12:19:57.012455+0000","last_clean":"2024-03-28T12:19:55.894779+0000","last_became_active":"2024-03-28T12:19:57.007966+0000","last_became_peered":"2024-03-28T12:19:57.007966+0000","last_unstale":"2024-03-28T12:19:57.012455+0000","last_undegraded":"2024-03-28T12:19:57.012455+0000","last_fullsized":"2024-03-28T12:19:57.012455+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_clean_scrub_stamp":"2024-03-28T12:19:55.894779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-03-28T12:19:56.986516+0000","last_change":"2024-03-28T12:19:56.986661+0000","last_active":"2024-03-28T12:19:56.986516+0000","last_peered":"2024-03-28T12:19:56.986516+0000","last_clean":"2024-03-28T12:19:55.894779+0000","last_became_active":"2024-03-28T12:19:56.981775+0000","last_became_peered":"2024-03-28T12:19:56.981775+0000","last_unstale":"2024-03-28T12:19:56.986516+0000","last_undegraded":"2024-03-28T12:19:56.986516+0000","last_fullsized":"2024-03-28T12:19:56.986516+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_clean_scrub_stamp":"2024-03-28T12:19:55.894779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":10,"state":"active+clean","last_fresh":"2024-03-28T12:19:56.980265+0000","last_change":"2024-03-28T12:19:56.980607+0000","last_active":"2024-03-28T12:19:56.980265+0000","last_peered":"2024-03-28T12:19:56.980265+0000","last_clean":"2024-03-28T12:19:55.894779+0000","last_became_active":"2024-03-28T12:19:56.975210+0000","last_became_peered":"2024-03-28T12:19:56.975210+0000","last_unstale":"2024-03-28T12:19:56.980265+0000","last_undegraded":"2024-03-28T12:19:56.980265+0000","last_fullsized":"2024-03-28T12:19:56.980265+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_clean_scrub_stamp":"2024-03-28T12:19:55.894779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-28T12:19:57.920420+0000","last_change":"2024-03-28T12:19:56.967505+0000","last_active":"2024-03-28T12:19:57.920420+0000","last_peered":"2024-03-28T12:19:57.920420+0000","last_clean":"2024-03-28T12:19:57.920420+0000","last_became_active":"2024-03-28T12:19:56.965117+0000","last_became_peered":"2024-03-28T12:19:56.965117+0000","last_unstale":"2024-03-28T12:19:57.920420+0000","last_undegraded":"2024-03-28T12:19:57.920420+0000","last_fullsized":"2024-03-28T12:19:57.920420+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_clean_scrub_stamp":"2024-03-28T12:19:55.894779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":138,"reported_epoch":11,"state":"active+clean","last_fresh":"2024-03-28T12:19:57.917612+0000","last_change":"2024-03-28T12:19:57.027039+0000","last_active":"2024-03-28T12:19:57.917612+0000","last_peered":"2024-03-28T12:19:57.917612+0000","last_clean":"2024-03-28T12:19:57.917612+0000","last_became_active":"2024-03-28T12:19:57.022989+0000","last_became_peered":"2024-03-28T12:19:57.022989+0000","last_unstale":"2024-03-28T12:19:57.917612+0000","last_undegraded":"2024-03-28T12:19:57.917612+0000","last_fullsized":"2024-03-28T12:19:57.917612+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-28T12:19:55.894779+0000","last_clean_scrub_stamp":"2024-03-28T12:19:55.894779+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26908,"kb_used_data":84,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344932,"statfs":{"total":96636764160,"available":96609210368,"internally_reserved":0,"allocated":86016,"data_stored":18958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26988,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344852,"statfs":{"total":96636764160,"available":96609128448,"internally_reserved":0,"allocated":155648,"data_stored":612589,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26980,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344860,"statfs":{"total":96636764160,"available":96609136640,"internally_reserved":0,"allocated":147456,"data_stored":609326,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-28T12:19:59.305 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-28T12:19:59.305 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-28T12:19:59.305 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-28T12:19:59.305 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-28T12:19:59.414 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:59.412+0000 7fe06179d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:59.416 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:19:59.416+0000 7fe06179d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:19:59.637 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:19:59.637 INFO:teuthology.orchestra.run.smithi183.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-28T12:19:59.650 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-28T12:19:59.651 INFO:teuthology.run_tasks:Running task cbt... 2024-03-28T12:19:59.667 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi183.front.sepia.ceph.com', 'osds': ['ubuntu@smithi183.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi183.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi183.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi183.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-03-28T12:19:59.667 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-03-28T12:19:59.685 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T12:19:59.685 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-03-28T12:19:59.737 DEBUG:teuthology.orchestra.run.smithi183:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-03-28T12:19:59.792 INFO:teuthology.orchestra.run.smithi183.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-03-28T12:20:00.196 INFO:tasks.cbt:Installing collectl 2024-03-28T12:20:00.197 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-03-28T12:20:00.297 INFO:teuthology.orchestra.run.smithi183.stderr:--2024-03-28 12:20:00-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-03-28T12:20:00.459 INFO:teuthology.orchestra.run.smithi183.stderr:Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::ac40:9691, ... 2024-03-28T12:20:00.469 INFO:teuthology.orchestra.run.smithi183.stderr:Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. 2024-03-28T12:20:00.685 INFO:teuthology.orchestra.run.smithi183.stderr:HTTP request sent, awaiting response... 302 Found 2024-03-28T12:20:00.686 INFO:teuthology.orchestra.run.smithi183.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABmBWBwYO-xZ5LvNh5ntxVCv2Tqt9UqZSS3uPP7u5BYCgFyVXqY9SB27h2Da9iKwbvyPeqFXyRtYmdrGIaIuqaULVzd_A%3D%3D&use_mirror=cfhcable&r= [following] 2024-03-28T12:20:00.686 INFO:teuthology.orchestra.run.smithi183.stderr:--2024-03-28 12:20:00-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABmBWBwYO-xZ5LvNh5ntxVCv2Tqt9UqZSS3uPP7u5BYCgFyVXqY9SB27h2Da9iKwbvyPeqFXyRtYmdrGIaIuqaULVzd_A%3D%3D&use_mirror=cfhcable&r= 2024-03-28T12:20:00.698 INFO:teuthology.orchestra.run.smithi183.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-03-28T12:20:00.766 INFO:teuthology.orchestra.run.smithi183.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-03-28T12:20:01.025 INFO:teuthology.orchestra.run.smithi183.stderr:HTTP request sent, awaiting response... 302 Found 2024-03-28T12:20:01.026 INFO:teuthology.orchestra.run.smithi183.stderr:Location: https://cfhcable.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2024-03-28T12:20:01.026 INFO:teuthology.orchestra.run.smithi183.stderr:--2024-03-28 12:20:01-- https://cfhcable.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2024-03-28T12:20:01.037 INFO:teuthology.orchestra.run.smithi183.stderr:Resolving cfhcable.dl.sourceforge.net (cfhcable.dl.sourceforge.net)... 146.71.73.6 2024-03-28T12:20:01.071 INFO:teuthology.orchestra.run.smithi183.stderr:Connecting to cfhcable.dl.sourceforge.net (cfhcable.dl.sourceforge.net)|146.71.73.6|:443... connected. 2024-03-28T12:20:01.244 INFO:teuthology.orchestra.run.smithi183.stderr:HTTP request sent, awaiting response... 200 OK 2024-03-28T12:20:01.244 INFO:teuthology.orchestra.run.smithi183.stderr:Length: 620693 (606K) [application/x-gzip] 2024-03-28T12:20:01.245 INFO:teuthology.orchestra.run.smithi183.stderr:Saving to: ‘collectl.tar.gz’ 2024-03-28T12:20:01.245 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-28T12:20:01.277 INFO:teuthology.orchestra.run.smithi183.stderr: 0K .......... .......... .......... .......... .......... 8% 1.46M 0s 2024-03-28T12:20:01.312 INFO:teuthology.orchestra.run.smithi183.stderr: 50K .......... .......... .......... .......... .......... 16% 1.45M 0s 2024-03-28T12:20:01.345 INFO:teuthology.orchestra.run.smithi183.stderr: 100K .......... .......... .......... .......... .......... 24% 1.51M 0s 2024-03-28T12:20:01.345 INFO:teuthology.orchestra.run.smithi183.stderr: 150K .......... .......... .......... .......... .......... 32% 63.2M 0s 2024-03-28T12:20:01.381 INFO:teuthology.orchestra.run.smithi183.stderr: 200K .......... .......... .......... .......... .......... 41% 1.53M 0s 2024-03-28T12:20:01.381 INFO:teuthology.orchestra.run.smithi183.stderr: 250K .......... .......... .......... .......... .......... 49% 95.7M 0s 2024-03-28T12:20:01.381 INFO:teuthology.orchestra.run.smithi183.stderr: 300K .......... .......... .......... .......... .......... 57% 104M 0s 2024-03-28T12:20:01.381 INFO:teuthology.orchestra.run.smithi183.stderr: 350K .......... .......... .......... .......... .......... 65% 70.5M 0s 2024-03-28T12:20:01.381 INFO:teuthology.orchestra.run.smithi183.stderr: 400K .......... .......... .......... .......... .......... 74% 107M 0s 2024-03-28T12:20:01.412 INFO:teuthology.orchestra.run.smithi183.stderr: 450K .......... .......... .......... .......... .......... 82% 1.57M 0s 2024-03-28T12:20:01.412 INFO:teuthology.orchestra.run.smithi183.stderr: 500K .......... .......... .......... .......... .......... 90% 70.7M 0s 2024-03-28T12:20:01.412 INFO:teuthology.orchestra.run.smithi183.stderr: 550K .......... .......... .......... .......... .......... 98% 89.1M 0s 2024-03-28T12:20:01.412 INFO:teuthology.orchestra.run.smithi183.stderr: 600K ...... 100% 11.4T=0.2s 2024-03-28T12:20:01.412 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-28T12:20:01.412 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28 12:20:01 (3.55 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-03-28T12:20:01.412 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-28T12:20:01.450 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/ 2024-03-28T12:20:01.451 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/collectl.conf 2024-03-28T12:20:01.451 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/ 2024-03-28T12:20:01.451 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/SlowProc.html 2024-03-28T12:20:01.451 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-03-28T12:20:01.451 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/WhySummary.html 2024-03-28T12:20:01.451 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Socket.html 2024-03-28T12:20:01.451 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/colmux.html 2024-03-28T12:20:01.451 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Tutorial.html 2024-03-28T12:20:01.451 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Gexpr.html 2024-03-28T12:20:01.451 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Import.html 2024-03-28T12:20:01.451 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/style.css 2024-03-28T12:20:01.451 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-03-28T12:20:01.452 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-03-28T12:20:01.452 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/FileNaming.html 2024-03-28T12:20:01.452 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/TheMath.html 2024-03-28T12:20:01.452 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Environmental.html 2024-03-28T12:20:01.452 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Process.html 2024-03-28T12:20:01.452 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-03-28T12:20:01.452 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/HiResTime.html 2024-03-28T12:20:01.452 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-03-28T12:20:01.452 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/slub.jpg 2024-03-28T12:20:01.452 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Lustre.html 2024-03-28T12:20:01.452 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Documentation.html 2024-03-28T12:20:01.452 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Inodes.html 2024-03-28T12:20:01.452 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Graphite.html 2024-03-28T12:20:01.453 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/InputFiles.html 2024-03-28T12:20:01.453 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Export.html 2024-03-28T12:20:01.453 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Interrupts.html 2024-03-28T12:20:01.453 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Startup.html 2024-03-28T12:20:01.453 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Performance.html 2024-03-28T12:20:01.453 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Messages.html 2024-03-28T12:20:01.453 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Logging.html 2024-03-28T12:20:01.453 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Data-detail.html 2024-03-28T12:20:01.453 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/CPUs.html 2024-03-28T12:20:01.453 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Matrix.html 2024-03-28T12:20:01.453 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Data-brief.html 2024-03-28T12:20:01.453 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-03-28T12:20:01.453 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Misc.html 2024-03-28T12:20:01.453 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Examples.html 2024-03-28T12:20:01.454 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/OpenStack.html 2024-03-28T12:20:01.454 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-03-28T12:20:01.454 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-03-28T12:20:01.454 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-03-28T12:20:01.454 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-03-28T12:20:01.454 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Data.html 2024-03-28T12:20:01.454 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Exceptions.html 2024-03-28T12:20:01.454 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-03-28T12:20:01.454 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-03-28T12:20:01.454 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-03-28T12:20:01.454 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-03-28T12:20:01.455 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Playback.html 2024-03-28T12:20:01.455 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Hello.html 2024-03-28T12:20:01.455 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Sockets.html 2024-03-28T12:20:01.455 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Network.html 2024-03-28T12:20:01.455 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Infiniband.html 2024-03-28T12:20:01.455 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Memory.html 2024-03-28T12:20:01.455 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-03-28T12:20:01.455 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/Disks.html 2024-03-28T12:20:01.455 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-03-28T12:20:01.455 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/service/ 2024-03-28T12:20:01.455 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/service/collectl.service 2024-03-28T12:20:01.455 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/initd/ 2024-03-28T12:20:01.455 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/initd/collectl-suse 2024-03-28T12:20:01.456 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/initd/collectl-debian 2024-03-28T12:20:01.456 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/initd/collectl-generic 2024-03-28T12:20:01.456 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/initd/collectl 2024-03-28T12:20:01.456 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/misc.ph 2024-03-28T12:20:01.456 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/gexpr.ph 2024-03-28T12:20:01.456 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/statsd.ph 2024-03-28T12:20:01.456 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/envrules.std 2024-03-28T12:20:01.456 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/hello.ph 2024-03-28T12:20:01.456 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/vmsum.ph 2024-03-28T12:20:01.456 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/proctree.ph 2024-03-28T12:20:01.456 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/README-WINDOWS 2024-03-28T12:20:01.456 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/formatit.ph 2024-03-28T12:20:01.456 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/colmux 2024-03-28T12:20:01.457 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/INSTALL 2024-03-28T12:20:01.457 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/graphite.ph 2024-03-28T12:20:01.457 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/RELEASE-collectl 2024-03-28T12:20:01.457 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/vmstat.ph 2024-03-28T12:20:01.457 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/README 2024-03-28T12:20:01.457 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/man1/ 2024-03-28T12:20:01.457 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/man1/collectl.1 2024-03-28T12:20:01.457 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/man1/colmux.1 2024-03-28T12:20:01.457 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/lexpr.ph 2024-03-28T12:20:01.457 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/client.pl 2024-03-28T12:20:01.457 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/collectl 2024-03-28T12:20:01.457 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/COPYING 2024-03-28T12:20:01.458 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/vnet.ph 2024-03-28T12:20:01.458 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/GPL 2024-03-28T12:20:01.458 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/UNINSTALL 2024-03-28T12:20:01.458 INFO:teuthology.orchestra.run.smithi183.stdout:collectl-4.3.1/ARTISTIC 2024-03-28T12:20:01.506 INFO:teuthology.orchestra.run.smithi183.stderr:./INSTALL: line 90: chkconfig: command not found 2024-03-28T12:20:01.509 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-03-28T12:20:02.085 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:03:04 ago on Thu 28 Mar 2024 12:16:58 PM UTC. 2024-03-28T12:20:02.186 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-03-28T12:20:02.188 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-03-28T12:20:02.245 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-28T12:20:02.247 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:20:02.247 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-28T12:20:02.247 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:20:02.247 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-28T12:20:02.247 INFO:teuthology.orchestra.run.smithi183.stdout: librbd-devel x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 23 k 2024-03-28T12:20:02.247 INFO:teuthology.orchestra.run.smithi183.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-03-28T12:20:02.247 INFO:teuthology.orchestra.run.smithi183.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-03-28T12:20:02.247 INFO:teuthology.orchestra.run.smithi183.stdout: perf x86_64 5.14.0-432.el9 appstream 2.0 M 2024-03-28T12:20:02.247 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-28T12:20:02.247 INFO:teuthology.orchestra.run.smithi183.stdout: libradospp-devel x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 30 k 2024-03-28T12:20:02.248 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:20:02.248 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-28T12:20:02.248 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-28T12:20:02.248 INFO:teuthology.orchestra.run.smithi183.stdout:Install 5 Packages 2024-03-28T12:20:02.248 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:20:02.248 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 2.2 M 2024-03-28T12:20:02.248 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 11 M 2024-03-28T12:20:02.248 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-28T12:20:02.840 INFO:teuthology.orchestra.run.smithi183.stdout:(1/5): libradospp-devel-19.0.0-2473.g785fc5e8.e 95 kB/s | 30 kB 00:00 2024-03-28T12:20:02.865 INFO:teuthology.orchestra.run.smithi183.stdout:(2/5): librbd-devel-19.0.0-2473.g785fc5e8.el9.x 67 kB/s | 23 kB 00:00 2024-03-28T12:20:03.115 INFO:teuthology.orchestra.run.smithi183.stdout:(3/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 52 kB/s | 13 kB 00:00 2024-03-28T12:20:03.307 INFO:teuthology.orchestra.run.smithi183.stdout:(4/5): pdsh-2.34-7.el9.x86_64.rpm 254 kB/s | 118 kB 00:00 2024-03-28T12:20:03.591 INFO:teuthology.orchestra.run.smithi183.stdout:(5/5): perf-5.14.0-432.el9.x86_64.rpm 1.9 MB/s | 2.0 MB 00:01 2024-03-28T12:20:03.594 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-28T12:20:03.595 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.6 MB/s | 2.2 MB 00:01 2024-03-28T12:20:03.622 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-28T12:20:03.642 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-28T12:20:03.642 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-28T12:20:03.759 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-28T12:20:03.764 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-28T12:20:04.061 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-28T12:20:04.135 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-03-28T12:20:04.173 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-03-28T12:20:04.205 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libradospp-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 3/5 2024-03-28T12:20:04.511 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librbd-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 4/5 2024-03-28T12:20:04.585 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : perf-5.14.0-432.el9.x86_64 5/5 2024-03-28T12:20:05.513 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: perf-5.14.0-432.el9.x86_64 5/5 2024-03-28T12:20:05.513 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libradospp-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/5 2024-03-28T12:20:05.513 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/5 2024-03-28T12:20:05.513 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : perf-5.14.0-432.el9.x86_64 3/5 2024-03-28T12:20:05.513 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-03-28T12:20:05.846 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-03-28T12:20:05.846 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:20:05.846 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-28T12:20:05.846 INFO:teuthology.orchestra.run.smithi183.stdout: libradospp-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:20:05.846 INFO:teuthology.orchestra.run.smithi183.stdout: librbd-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-28T12:20:05.847 INFO:teuthology.orchestra.run.smithi183.stdout: pdsh-2.34-7.el9.x86_64 2024-03-28T12:20:05.847 INFO:teuthology.orchestra.run.smithi183.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-03-28T12:20:05.847 INFO:teuthology.orchestra.run.smithi183.stdout: perf-5.14.0-432.el9.x86_64 2024-03-28T12:20:05.847 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-28T12:20:05.847 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-28T12:20:06.184 INFO:tasks.cbt:benchmark: radosbench 2024-03-28T12:20:06.184 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-03-28T12:20:06.214 DEBUG:teuthology.orchestra.run.smithi183:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-03-28T12:20:06.606 INFO:teuthology.orchestra.run.smithi183.stderr:12:20:06 - DEBUG - cbt - Settings.cluster: 2024-03-28T12:20:06.606 INFO:teuthology.orchestra.run.smithi183.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-03-28T12:20:06.606 INFO:teuthology.orchestra.run.smithi183.stderr: 'clients': ['ubuntu@smithi183.front.sepia.ceph.com'], 2024-03-28T12:20:06.606 INFO:teuthology.orchestra.run.smithi183.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-03-28T12:20:06.607 INFO:teuthology.orchestra.run.smithi183.stderr: 'head': 'ubuntu@smithi183.front.sepia.ceph.com', 2024-03-28T12:20:06.607 INFO:teuthology.orchestra.run.smithi183.stderr: 'is_teuthology': True, 2024-03-28T12:20:06.607 INFO:teuthology.orchestra.run.smithi183.stderr: 'iterations': 1, 2024-03-28T12:20:06.607 INFO:teuthology.orchestra.run.smithi183.stderr: 'mons': ['ubuntu@smithi183.front.sepia.ceph.com'], 2024-03-28T12:20:06.607 INFO:teuthology.orchestra.run.smithi183.stderr: 'osds': ['ubuntu@smithi183.front.sepia.ceph.com'], 2024-03-28T12:20:06.607 INFO:teuthology.orchestra.run.smithi183.stderr: 'osds_per_node': 3, 2024-03-28T12:20:06.607 INFO:teuthology.orchestra.run.smithi183.stderr: 'pid_dir': '/var/run/ceph', 2024-03-28T12:20:06.607 INFO:teuthology.orchestra.run.smithi183.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2024-03-28T12:20:06.607 INFO:teuthology.orchestra.run.smithi183.stderr: 'pgp_size': 256, 2024-03-28T12:20:06.607 INFO:teuthology.orchestra.run.smithi183.stderr: 'replication': 'replicated'}}, 2024-03-28T12:20:06.607 INFO:teuthology.orchestra.run.smithi183.stderr: 'rebuild_every_test': False, 2024-03-28T12:20:06.608 INFO:teuthology.orchestra.run.smithi183.stderr: 'rgws': {'ubuntu@smithi183.front.sepia.ceph.com': None}, 2024-03-28T12:20:06.608 INFO:teuthology.orchestra.run.smithi183.stderr: 'tmp_dir': '/tmp/cbt', 2024-03-28T12:20:06.608 INFO:teuthology.orchestra.run.smithi183.stderr: 'use_existing': True, 2024-03-28T12:20:06.608 INFO:teuthology.orchestra.run.smithi183.stderr: 'user': 'ubuntu'} 2024-03-28T12:20:06.626 INFO:teuthology.orchestra.run.smithi183.stderr:12:20:06 - INFO - cbt - Pausing for 60s for idle monitoring. 2024-03-28T12:20:06.628 INFO:teuthology.orchestra.run.smithi183.stderr:12:20:06 - DEBUG - cbt - Nodes : ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:20:06.628 INFO:teuthology.orchestra.run.smithi183.stderr:12:20:06 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:20:06.628 INFO:teuthology.orchestra.run.smithi183.stderr:12:20:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl 2024-03-28T12:20:06.631 INFO:teuthology.orchestra.run.smithi183.stderr:12:20:06 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:20:06.631 INFO:teuthology.orchestra.run.smithi183.stderr:12:20:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-03-28T12:20:06.633 INFO:teuthology.orchestra.run.smithi183.stderr:12:20:06 - DEBUG - cbt - Nodes : ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:20:06.633 INFO:teuthology.orchestra.run.smithi183.stderr:12:20:06 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:20:06.633 INFO:teuthology.orchestra.run.smithi183.stderr:12:20:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf 2024-03-28T12:20:06.638 INFO:teuthology.orchestra.run.smithi183.stderr:12:20:06 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:20:06.638 INFO:teuthology.orchestra.run.smithi183.stderr:12:20:06 - DEBUG - cbt - PerfMonitoring: in local_node 2024-03-28T12:20:06.638 INFO:teuthology.orchestra.run.smithi183.stderr:12:20:06 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-03-28T12:21:06.685 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:06.686 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-03-28T12:21:06.729 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:06.729 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-03-28T12:21:06.772 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:06.772 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2024-03-28T12:21:06.774 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:06.774 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2024-03-28T12:21:06.776 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - Nodes : ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:21:06.776 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:06.776 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* 2024-03-28T12:21:06.792 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:06.792 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/$(basename ${i}).smithi183.front.sepia.ceph.com ; done 2024-03-28T12:21:06.797 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - Nodes : ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:21:06.797 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:06.797 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo killall -9 rados 2024-03-28T12:21:06.827 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - WARNING - cbt - sudo killall -9 rados 2024-03-28T12:21:06.827 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - WARNING - cbt - error 1 seen, continuing anyway... 2024-03-28T12:21:06.846 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3. 2024-03-28T12:21:06.846 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2024-03-28T12:21:06.846 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - Nodes : ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:21:06.846 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:06.846 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2024-03-28T12:21:06.862 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - Nodes : ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:21:06.862 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:06.862 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl 2024-03-28T12:21:06.864 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:06.865 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-03-28T12:21:06.866 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - Nodes : ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:21:06.866 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:06.866 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf 2024-03-28T12:21:06.869 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:06.869 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - PerfMonitoring: in local_node 2024-03-28T12:21:06.869 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-03-28T12:21:06.869 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - Nodes : ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:21:06.870 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:06.870 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:06 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2024-03-28T12:21:07.182 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:07 - DEBUG - cbt - Nodes : ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:21:07.182 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:07 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:07.182 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:07 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2024-03-28T12:21:07.980 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:07 - DEBUG - cbt - Nodes : ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:21:07.980 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:07 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:07.980 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:07 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2024-03-28T12:21:08.989 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:08 - INFO - cbt - Checking Health after pool creation. 2024-03-28T12:21:13.991 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:13 - DEBUG - cbt - Nodes : ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:21:13.991 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:13 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:13.991 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-03-28T12:21:14.346 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:14 - INFO - cbt - Final Pool Health Check. 2024-03-28T12:21:19.352 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:19 - DEBUG - cbt - Nodes : ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:21:19.352 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:19 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:19.352 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-03-28T12:21:19.688 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:19 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:19.688 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-03-28T12:21:19.713 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:19 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:19.713 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-03-28T12:21:19.752 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:19 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:19.752 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2024-03-28T12:21:19.754 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:19 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:19.754 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2024-03-28T12:21:19.756 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:19 - DEBUG - cbt - Nodes : ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:21:19.756 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:19 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:19.756 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-03-28T12:21:19.872 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:19 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:19.872 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-03-28T12:21:20.408 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:20 - DEBUG - cbt - Nodes : ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:21:20.408 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:20 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:20.408 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2024-03-28T12:21:20.746 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:20 - DEBUG - cbt - Nodes : ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:21:20.747 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:20 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:20.747 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:20 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write 2024-03-28T12:21:20.813 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:20 - DEBUG - cbt - Nodes : ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:21:20.813 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:20 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:20.813 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/ceph_settings.out 2024-03-28T12:21:21.642 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:21 - DEBUG - cbt - Nodes : ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:21:21.642 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:21 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:21.642 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl 2024-03-28T12:21:21.644 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:21 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:21.645 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-03-28T12:21:21.646 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:21 - DEBUG - cbt - Nodes : ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:21:21.646 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:21 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:21.646 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf 2024-03-28T12:21:21.649 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:21 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:21.649 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:21 - DEBUG - cbt - PerfMonitoring: in local_node 2024-03-28T12:21:21.650 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:21 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-03-28T12:21:21.650 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:21 - INFO - cbt - Running radosbench write test. 2024-03-28T12:21:21.650 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:21 - DEBUG - cbt - Nodes : ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:21:21.650 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:21 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:21.650 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.0 2024-03-28T12:21:21.651 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:21 - DEBUG - cbt - Nodes : ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:21:21.651 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:21 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:21:21.651 INFO:teuthology.orchestra.run.smithi183.stderr:12:21:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.1 2024-03-28T12:22:22.800 INFO:teuthology.orchestra.run.smithi183.stderr:12:22:22 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:22:22.801 INFO:teuthology.orchestra.run.smithi183.stderr:12:22:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-03-28T12:22:22.896 INFO:teuthology.orchestra.run.smithi183.stderr:12:22:22 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:22:22.896 INFO:teuthology.orchestra.run.smithi183.stderr:12:22:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-03-28T12:22:22.935 INFO:teuthology.orchestra.run.smithi183.stderr:12:22:22 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:22:22.935 INFO:teuthology.orchestra.run.smithi183.stderr:12:22:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf.data 2024-03-28T12:22:22.937 INFO:teuthology.orchestra.run.smithi183.stderr:12:22:22 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:22:22.937 INFO:teuthology.orchestra.run.smithi183.stderr:12:22:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf_stat.* 2024-03-28T12:22:22.939 INFO:teuthology.orchestra.run.smithi183.stderr:12:22:22 - DEBUG - cbt - Nodes : ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:22:22.939 INFO:teuthology.orchestra.run.smithi183.stderr:12:22:22 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:22:22.939 INFO:teuthology.orchestra.run.smithi183.stderr:12:22:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/historic_ops.out 2024-03-28T12:22:23.251 INFO:teuthology.orchestra.run.smithi183.stderr:12:22:23 - DEBUG - cbt - Nodes : ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:22:23.264 INFO:teuthology.orchestra.run.smithi183.stderr:12:22:23 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:22:23.264 INFO:teuthology.orchestra.run.smithi183.stderr:12:22:23 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* 2024-03-28T12:22:23.280 INFO:teuthology.orchestra.run.smithi183.stderr:12:22:23 - DEBUG - cbt - get_fqdn_local()=smithi183.front.sepia.ceph.com 2024-03-28T12:22:23.281 INFO:teuthology.orchestra.run.smithi183.stderr:12:22:23 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/$(basename ${i}).smithi183.front.sepia.ceph.com ; done 2024-03-28T12:22:23.312 INFO:teuthology.orchestra.run.smithi183.stderr:12:22:23 - INFO - cbt - Convert results to json format. 2024-03-28T12:22:23.312 INFO:teuthology.orchestra.run.smithi183.stderr:12:22:23 - DEBUG - cbt - Nodes : ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:22:23.331 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2024-03-28T12:22:23.340 DEBUG:teuthology.orchestra.run.smithi183:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2024-03-28T12:22:23.530 INFO:tasks.cbt_performance:Collecting CBT performance data config 2024-03-28T12:22:23.530 INFO:tasks.cbt_performance:reading cbt preferences 2024-03-28T12:22:23.531 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -name 'json_output.*' 2024-03-28T12:22:23.592 INFO:teuthology.orchestra.run.smithi183.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.1.smithi183.front.sepia.ceph.com 2024-03-28T12:22:23.592 INFO:teuthology.orchestra.run.smithi183.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.0.smithi183.front.sepia.ceph.com 2024-03-28T12:22:23.592 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T12:22:23.593 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.1.smithi183.front.sepia.ceph.com of=/dev/stdout 2024-03-28T12:22:23.609 INFO:tasks.cbt_performance:reading total cpu cycles 2024-03-28T12:22:23.609 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write -name 'perf_stat.*' 2024-03-28T12:22:23.664 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-03-28T12:22:23.664 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-03-28T12:22:23.664 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-28T12:22:23.664 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.0.smithi183.front.sepia.ceph.com of=/dev/stdout 2024-03-28T12:22:23.719 INFO:tasks.cbt_performance:reading total cpu cycles 2024-03-28T12:22:23.719 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write -name 'perf_stat.*' 2024-03-28T12:22:23.773 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-03-28T12:22:23.773 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-03-28T12:22:23.782 INFO:tasks.cbt_performance:Data inserted successfully. 2024-03-28T12:22:23.788 INFO:tasks.cbt_performance:Data inserted successfully. 2024-03-28T12:22:23.788 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-28T12:22:23.921 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-28T12:22:24.046 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:22:24.040+0000 7fd3e435d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:22:24.048 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-28T12:22:24.049+0000 7fd3e435d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-28T12:22:24.251 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-28T12:22:24.251 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-28T12:22:24.251 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-28T12:22:24.251 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T12:22:24.347 INFO:tasks.ceph.osd.0:Stopped 2024-03-28T12:22:24.347 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-28T12:22:24.347 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T12:22:24.439 INFO:tasks.ceph.osd.1:Stopped 2024-03-28T12:22:24.439 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-28T12:22:24.439 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T12:22:24.522 INFO:tasks.ceph.osd.2:Stopped 2024-03-28T12:22:24.522 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-28T12:22:24.522 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-28T12:22:24.522 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T12:22:24.549 INFO:tasks.ceph.mgr.x:Stopped 2024-03-28T12:22:24.550 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-28T12:22:24.550 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-28T12:22:24.550 INFO:teuthology.orchestra.run:waiting for 300 2024-03-28T12:22:24.600 INFO:tasks.ceph.mon.a:Stopped 2024-03-28T12:22:24.601 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-28T12:22:24.601 DEBUG:teuthology.orchestra.run.smithi183:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v 'overall HEALTH' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-28T12:22:24.685 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:22:24.685 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-28T12:22:25.449 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:22:25.449 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-28T12:22:25.550 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:22:25.550 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-28T12:22:25.603 INFO:tasks.ceph:Archiving mon data... 2024-03-28T12:22:25.604 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627765/data/mon.a.tgz 2024-03-28T12:22:25.605 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-03-28T12:22:25.624 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.Rn7G2nAMNg 2024-03-28T12:22:25.624 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Rn7G2nAMNg 2024-03-28T12:22:25.754 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.Rn7G2nAMNg 2024-03-28T12:22:25.863 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.Rn7G2nAMNg is 180KB 2024-03-28T12:22:25.914 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.Rn7G2nAMNg 2024-03-28T12:22:25.928 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-28T12:22:25.928 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-28T12:22:26.055 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-28T12:22:26.055 INFO:tasks.ceph:Archiving crash dumps... 2024-03-28T12:22:26.056 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627765/remote/smithi183/crash 2024-03-28T12:22:26.057 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-28T12:22:26.121 INFO:tasks.ceph:Compressing logs... 2024-03-28T12:22:26.121 DEBUG:teuthology.orchestra.run.smithi183:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-28T12:22:26.196 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-28T12:22:26.196 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30431.log 2024-03-28T12:22:26.196 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30868.log 2024-03-28T12:22:26.206 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.30431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31319.log 2024-03-28T12:22:26.206 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.30868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32390.log 2024-03-28T12:22:26.206 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.31319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31489.log 2024-03-28T12:22:26.207 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.32390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32516.log 2024-03-28T12:22:26.207 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.31489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31318.log 2024-03-28T12:22:26.207 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.32516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31405.log 2024-03-28T12:22:26.207 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.31318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32455.log 2024-03-28T12:22:26.207 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.31405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32492.log 2024-03-28T12:22:26.207 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.32455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29995.log 2024-03-28T12:22:26.207 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.32492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32539.log 2024-03-28T12:22:26.207 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.29995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31193.log 2024-03-28T12:22:26.207 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.32539.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-28T12:22:26.207 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.31193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30233.log 2024-03-28T12:22:26.207 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28636.log 2024-03-28T12:22:26.207 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.30233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30083.log 2024-03-28T12:22:26.207 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph.tmp-client.admin.28636.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-28T12:22:26.207 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.30083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31192.log 2024-03-28T12:22:26.207 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30924.log 2024-03-28T12:22:26.207 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.1.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.29995.log.gz 2024-03-28T12:22:26.207 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.31192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30966.log 2024-03-28T12:22:26.208 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30193.log 2024-03-28T12:22:26.208 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.30924.log: 66.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28636.log.gz 2024-03-28T12:22:26.208 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.30966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30461.log 2024-03-28T12:22:26.208 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.30193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31008.log 2024-03-28T12:22:26.208 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.30461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31279.log 2024-03-28T12:22:26.208 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.31008.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-28T12:22:26.208 INFO:teuthology.orchestra.run.smithi183.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30083.log.gz 2024-03-28T12:22:26.208 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.31279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32897.log 2024-03-28T12:22:26.208 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32603.log 2024-03-28T12:22:26.208 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.32897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31447.log 2024-03-28T12:22:26.208 INFO:teuthology.orchestra.run.smithi183.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.32897.log.gz 2024-03-28T12:22:26.208 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.32603.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-28T12:22:26.208 INFO:teuthology.orchestra.run.smithi183.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.32603.log.gz 2024-03-28T12:22:26.208 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.31447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30273.log 2024-03-28T12:22:26.208 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31051.log 2024-03-28T12:22:26.209 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.30273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32604.log 2024-03-28T12:22:26.209 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.31051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31191.log 2024-03-28T12:22:26.209 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.32604.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-28T12:22:26.209 INFO:teuthology.orchestra.run.smithi183.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.32604.log.gz 2024-03-28T12:22:26.209 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.31191.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-28T12:22:26.209 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31111.log 2024-03-28T12:22:26.209 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.2.log: 85.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-28T12:22:26.209 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.31111.log: 90.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-28T12:22:26.212 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30431.log.gz 2024-03-28T12:22:26.216 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30461.log.gz 2024-03-28T12:22:26.216 INFO:teuthology.orchestra.run.smithi183.stderr: 87.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-28T12:22:26.223 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30868.log.gz 2024-03-28T12:22:26.223 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30924.log.gz 2024-03-28T12:22:26.223 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30966.log.gz 2024-03-28T12:22:26.223 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31008.log.gz 2024-03-28T12:22:26.223 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31051.log.gz 2024-03-28T12:22:26.223 INFO:teuthology.orchestra.run.smithi183.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31111.log.gz 2024-03-28T12:22:26.223 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31193.log.gz 2024-03-28T12:22:26.223 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31191.log.gz 2024-03-28T12:22:26.223 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31192.log.gz 2024-03-28T12:22:26.223 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31279.log.gz 2024-03-28T12:22:26.223 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31318.log.gz 2024-03-28T12:22:26.223 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31319.log.gz 2024-03-28T12:22:26.223 INFO:teuthology.orchestra.run.smithi183.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.31405.log.gz 2024-03-28T12:22:26.223 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31447.log.gz 2024-03-28T12:22:26.224 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31489.log.gz 2024-03-28T12:22:26.255 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30193.log.gz 2024-03-28T12:22:26.255 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30233.log.gz 2024-03-28T12:22:26.256 INFO:teuthology.orchestra.run.smithi183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30273.log.gz 2024-03-28T12:22:26.280 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32539.log.gz 2024-03-28T12:22:26.280 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32516.log.gz 2024-03-28T12:22:26.316 INFO:teuthology.orchestra.run.smithi183.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-28T12:22:26.341 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32390.log.gz 2024-03-28T12:22:26.341 INFO:teuthology.orchestra.run.smithi183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32455.log.gz 2024-03-28T12:22:26.341 INFO:teuthology.orchestra.run.smithi183.stderr: 48.0% -- replaced with /var/log/ceph/ceph-client.admin.32492.log.gz 2024-03-28T12:22:27.892 INFO:teuthology.orchestra.run.smithi183.stderr: 94.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-28T12:22:27.977 INFO:teuthology.orchestra.run.smithi183.stderr: 94.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-28T12:22:28.119 INFO:teuthology.orchestra.run.smithi183.stderr: 94.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-28T12:22:28.121 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-28T12:22:28.122 INFO:teuthology.orchestra.run.smithi183.stderr:real 0m1.946s 2024-03-28T12:22:28.122 INFO:teuthology.orchestra.run.smithi183.stderr:user 0m4.210s 2024-03-28T12:22:28.122 INFO:teuthology.orchestra.run.smithi183.stderr:sys 0m0.210s 2024-03-28T12:22:28.122 INFO:tasks.ceph:Archiving logs... 2024-03-28T12:22:28.122 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/log/ceph to /home/teuthworker/archive/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627765/remote/smithi183/log 2024-03-28T12:22:28.123 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-28T12:22:28.802 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-28T12:22:28.812 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-28T12:22:28.812 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-28T12:22:28.887 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-28T12:22:28.896 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-28T12:22:28.896 DEBUG:teuthology.orchestra.run.smithi183:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-28T12:22:28.913 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-03-28T12:22:28.978 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-28T12:22:28.978 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-03-28T12:22:28.978 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 20 +344us[ +344us] +/- 75ms 2024-03-28T12:22:28.979 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 19 -963us[ -963us] +/- 87ms 2024-03-28T12:22:28.979 INFO:teuthology.orchestra.run.smithi183.stdout:^* hv03.front.sepia.ceph.com 3 6 377 21 +484us[ +494us] +/- 68ms 2024-03-28T12:22:28.979 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-28T12:22:28.985 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-28T12:22:28.994 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-28T12:22:28.995 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-28T12:22:29.003 DEBUG:teuthology.orchestra.run.smithi183:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-28T12:22:29.148 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-28T12:22:29.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T12:22:29.178 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-28T12:22:29.187 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-28T12:22:29.196 INFO:teuthology.task.internal:Duration was 604.329306 seconds 2024-03-28T12:22:29.196 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-28T12:22:29.205 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-28T12:22:29.205 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-28T12:22:29.351 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-28T12:22:29.901 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-28T12:22:29.901 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi183.front.sepia.ceph.com 2024-03-28T12:22:29.901 DEBUG:teuthology.orchestra.run.smithi183:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-28T12:22:29.921 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-28T12:22:29.921 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-28T12:22:29.990 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-28T12:22:29.990 DEBUG:teuthology.orchestra.run.smithi183:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-28T12:22:30.137 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-28T12:22:30.147 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-28T12:22:30.147 DEBUG:teuthology.orchestra.run.smithi183:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-28T12:22:30.183 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-28T12:22:30.193 DEBUG:teuthology.orchestra.run.smithi183:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-28T12:22:30.263 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = core 2024-03-28T12:22:30.306 DEBUG:teuthology.orchestra.run.smithi183:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-28T12:22:30.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-28T12:22:30.360 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-28T12:22:30.369 INFO:teuthology.task.internal:Transferring archived files... 2024-03-28T12:22:30.370 DEBUG:teuthology.misc:Transferring archived files from smithi183:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yingxin-2024-03-28_05:47:02-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7627765/remote/smithi183 2024-03-28T12:22:30.370 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-28T12:22:30.465 INFO:teuthology.task.internal:Removing archive directory... 2024-03-28T12:22:30.465 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-28T12:22:30.520 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-28T12:22:30.533 INFO:teuthology.task.internal:Not uploading archives. 2024-03-28T12:22:30.533 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-28T12:22:30.543 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-28T12:22:30.543 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-28T12:22:30.575 INFO:teuthology.orchestra.run.smithi183.stdout: 265594 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 28 12:22 /home/ubuntu/cephtest 2024-03-28T12:22:30.576 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-28T12:22:30.586 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-28T12:22:30.607 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/Rfy7LkB4k/rados-bench?orgId=1&var-branch_name=ci-yingxin-crimson-load-obc-3&var-job_id_selected=7627765 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} duration: 604.3293056488037 flavor: crimson owner: scheduled_yingxin@teuthology success: true 2024-03-28T12:22:30.609 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-28T12:22:30.699 INFO:teuthology.run:pass