2023-11-27T09:08:41.201 INFO:root:teuthology version: 0.0.1.dev205+g2442239 2023-11-27T09:08:41.201 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467536 --owner scheduled_yingxin@teuthology --verbose --name yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_rand_read} -- /home/teuthworker/archive/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467536/orig.config.yaml 2023-11-27T09:08:41.243 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-27T09:08:41.316 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467536 branch: wip-yingxin-crimson-make-crosscore-send-ordered description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_rand_read} email: yingxin.cheng@intel.com first_in_suite: false job_id: '7467536' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yingxin-crimson-make-crosscore-send-ordered ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: 705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: 705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3 workunit: branch: wip-yingxin-crimson-make-crosscore-send-ordered env: CRIMSON_COMPAT: '1' sha1: 705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3 owner: scheduled_yingxin@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3 sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-yingxin-crimson-make-crosscore-send-ordered suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3 targets: smithi007.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIo1oSgQ96OcqSO1+jWyD2Ss+I4+QETc12xGTgB8SwCMRoGMHUTdGkPL3ekuNjOA5AelGYNkpwSR4suTNnzD0VY= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4096 pool_monitoring_list: - collectl pool_profile: replicated readmode: rand run_monitoring_list: - collectl time: 60 write_only: false cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 2442239d2653456406c25ae0c71b689c8f2657b6 timestamp: 2023-11-27_09:03:31 tube: smithi user: yingxin verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-11-27T09:08:41.316 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3/qa; will attempt to use it 2023-11-27T09:08:41.317 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3/qa/tasks 2023-11-27T09:08:41.317 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-27T09:08:41.318 INFO:teuthology.task.internal:Checking packages... 2023-11-27T09:08:41.340 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3' 2023-11-27T09:08:41.340 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-27T09:08:41.340 INFO:teuthology.packaging:ref: None 2023-11-27T09:08:41.340 INFO:teuthology.packaging:tag: None 2023-11-27T09:08:41.340 INFO:teuthology.packaging:branch: wip-yingxin-crimson-make-crosscore-send-ordered 2023-11-27T09:08:41.340 INFO:teuthology.packaging:sha1: 705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3 2023-11-27T09:08:41.340 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-yingxin-crimson-make-crosscore-send-ordered 2023-11-27T09:08:41.556 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7476.g705aae7f 2023-11-27T09:08:41.556 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-27T09:08:41.563 INFO:teuthology.task.internal:no buildpackages task found 2023-11-27T09:08:41.563 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-27T09:08:41.575 INFO:teuthology.task.internal:Saving configuration 2023-11-27T09:08:41.590 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-27T09:08:41.596 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-27T09:08:41.623 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467536', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-27 09:03:57.533502', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIo1oSgQ96OcqSO1+jWyD2Ss+I4+QETc12xGTgB8SwCMRoGMHUTdGkPL3ekuNjOA5AelGYNkpwSR4suTNnzD0VY='} 2023-11-27T09:08:41.623 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-27T09:08:41.629 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-11-27T09:08:41.630 INFO:teuthology.run_tasks:Running task console_log... 2023-11-27T09:08:41.667 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4f1e90ff70>, signals=[15]) 2023-11-27T09:08:41.667 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-27T09:08:41.674 INFO:teuthology.task.internal:Opening connections... 2023-11-27T09:08:41.674 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:08:41.675 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-27T09:08:41.774 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-27T09:08:41.780 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2023-11-27T09:08:41.843 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2023-11-27T09:08:41.843 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2023-11-27T09:08:41.900 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="CentOS Stream" 2023-11-27T09:08:41.900 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="9" 2023-11-27T09:08:41.900 INFO:teuthology.orchestra.run.smithi007.stdout:ID="centos" 2023-11-27T09:08:41.900 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE="rhel fedora" 2023-11-27T09:08:41.900 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="9" 2023-11-27T09:08:41.900 INFO:teuthology.orchestra.run.smithi007.stdout:PLATFORM_ID="platform:el9" 2023-11-27T09:08:41.900 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-27T09:08:41.900 INFO:teuthology.orchestra.run.smithi007.stdout:ANSI_COLOR="0;31" 2023-11-27T09:08:41.900 INFO:teuthology.orchestra.run.smithi007.stdout:LOGO="fedora-logo-icon" 2023-11-27T09:08:41.900 INFO:teuthology.orchestra.run.smithi007.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-27T09:08:41.900 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://centos.org/" 2023-11-27T09:08:41.901 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-27T09:08:41.901 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-27T09:08:41.901 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-27T09:08:41.901 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2023-11-27T09:08:41.926 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-27T09:08:41.935 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-27T09:08:41.940 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-27T09:08:41.940 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2023-11-27T09:08:41.956 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-27T09:08:41.963 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-27T09:08:41.963 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2023-11-27T09:08:42.017 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-27T09:08:42.049 INFO:teuthology.run_tasks:Running task kernel... 2023-11-27T09:08:42.062 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-27T09:08:42.063 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-27T09:08:42.063 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-27T09:08:42.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-27T09:08:42.079 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2023-11-27T09:08:42.135 INFO:teuthology.orchestra.run.smithi007.stdout:5.14.0-361.el9.x86_64 2023-11-27T09:08:42.135 INFO:teuthology.task.kernel:Running kernel on smithi007: 5.14.0-361.el9.x86_64 2023-11-27T09:08:42.136 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum install -y kernel 2023-11-27T09:08:44.182 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - AppStream 15 MB/s | 6.4 MB 00:00 2023-11-27T09:08:45.557 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2023-11-27T09:08:47.548 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 7.9 MB 00:01 2023-11-27T09:08:51.007 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - AppStream 7.9 MB/s | 18 MB 00:02 2023-11-27T09:08:55.024 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - CRB 12 MB/s | 5.8 MB 00:00 2023-11-27T09:08:56.534 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 15 kB 00:00 2023-11-27T09:08:57.552 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 23 MB/s | 20 MB 00:00 2023-11-27T09:09:02.477 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2023-11-27T09:09:04.950 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-11-27T09:09:04.951 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-11-27T09:09:05.040 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:09:05.055 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:09:05.055 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2023-11-27T09:09:05.056 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:09:05.056 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-27T09:09:05.056 INFO:teuthology.orchestra.run.smithi007.stdout: kernel x86_64 5.14.0-386.el9 baseos 5.5 M 2023-11-27T09:09:05.056 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-11-27T09:09:05.056 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-core x86_64 5.14.0-386.el9 baseos 20 M 2023-11-27T09:09:05.056 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules x86_64 5.14.0-386.el9 baseos 38 M 2023-11-27T09:09:05.056 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-core x86_64 5.14.0-386.el9 baseos 32 M 2023-11-27T09:09:05.056 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:09:05.057 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-27T09:09:05.057 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:09:05.057 INFO:teuthology.orchestra.run.smithi007.stdout:Install 4 Packages 2023-11-27T09:09:05.057 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:09:05.058 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 95 M 2023-11-27T09:09:05.058 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 121 M 2023-11-27T09:09:05.058 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-27T09:09:05.903 INFO:teuthology.orchestra.run.smithi007.stdout:(1/4): kernel-5.14.0-386.el9.x86_64.rpm 15 MB/s | 5.5 MB 00:00 2023-11-27T09:09:06.553 INFO:teuthology.orchestra.run.smithi007.stdout:(2/4): kernel-core-5.14.0-386.el9.x86_64.rpm 20 MB/s | 20 MB 00:01 2023-11-27T09:09:07.245 INFO:teuthology.orchestra.run.smithi007.stdout:(3/4): kernel-modules-5.14.0-386.el9.x86_64.rpm 22 MB/s | 38 MB 00:01 2023-11-27T09:09:08.537 INFO:teuthology.orchestra.run.smithi007.stdout:(4/4): kernel-modules-core-5.14.0-386.el9.x86_6 12 MB/s | 32 MB 00:02 2023-11-27T09:09:08.537 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-27T09:09:08.538 INFO:teuthology.orchestra.run.smithi007.stdout:Total 27 MB/s | 95 MB 00:03 2023-11-27T09:09:09.419 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-27T09:09:09.834 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-27T09:09:09.834 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-27T09:09:11.215 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-27T09:09:11.215 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-27T09:09:13.992 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-27T09:09:14.678 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-modules-core-5.14.0-386.el9.x86_64 1/4 2023-11-27T09:09:14.798 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-core-5.14.0-386.el9.x86_64 2/4 2023-11-27T09:09:17.695 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-core-5.14.0-386.el9.x86_64 2/4 2023-11-27T09:09:17.876 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-modules-5.14.0-386.el9.x86_64 3/4 2023-11-27T09:09:22.831 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-5.14.0-386.el9.x86_64 3/4 2023-11-27T09:09:22.951 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-5.14.0-386.el9.x86_64 4/4 2023-11-27T09:09:27.930 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-core-5.14.0-386.el9.x86_64 4/4 2023-11-27T09:09:58.428 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-core-5.14.0-386.el9.x86_64 4/4 2023-11-27T09:09:58.473 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-5.14.0-386.el9.x86_64 4/4 2023-11-27T09:09:58.946 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-5.14.0-386.el9.x86_64 4/4 2023-11-27T09:09:58.946 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-5.14.0-386.el9.x86_64 1/4 2023-11-27T09:09:58.946 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-core-5.14.0-386.el9.x86_64 2/4 2023-11-27T09:09:58.946 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-modules-5.14.0-386.el9.x86_64 3/4 2023-11-27T09:09:59.360 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-modules-core-5.14.0-386.el9.x86_64 4/4 2023-11-27T09:09:59.360 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:09:59.360 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-27T09:09:59.361 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-5.14.0-386.el9.x86_64 2023-11-27T09:09:59.361 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-core-5.14.0-386.el9.x86_64 2023-11-27T09:09:59.361 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-5.14.0-386.el9.x86_64 2023-11-27T09:09:59.361 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-core-5.14.0-386.el9.x86_64 2023-11-27T09:09:59.361 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:09:59.361 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:09:59.853 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2023-11-27T09:10:00.017 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-5.14.0-386.el9.x86_64 2023-11-27T09:10:00.017 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-386.el9.x86_64 2023-11-27T09:10:00.017 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-386.el9.x86_64 2023-11-27T09:10:00.017 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-27T09:10:00.017 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-27T09:10:00.017 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-11-27T09:10:00.018 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi007.front.sepia.ceph.com, path=None, version=distro) 2023-11-27T09:10:00.018 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2023-11-27T09:10:00.139 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-5.14.0-386.el9.x86_64 2023-11-27T09:10:00.140 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-386.el9.x86_64 2023-11-27T09:10:00.140 DEBUG:teuthology.orchestra.run.smithi007:> sudo rpm -qi grub2-tools 2023-11-27T09:10:00.178 INFO:teuthology.orchestra.run.smithi007.stdout:Name : grub2-tools 2023-11-27T09:10:00.178 INFO:teuthology.orchestra.run.smithi007.stdout:Epoch : 1 2023-11-27T09:10:00.178 INFO:teuthology.orchestra.run.smithi007.stdout:Version : 2.06 2023-11-27T09:10:00.178 INFO:teuthology.orchestra.run.smithi007.stdout:Release : 68.el9 2023-11-27T09:10:00.178 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: x86_64 2023-11-27T09:10:00.178 INFO:teuthology.orchestra.run.smithi007.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-11-27T09:10:00.178 INFO:teuthology.orchestra.run.smithi007.stdout:Group : Unspecified 2023-11-27T09:10:00.178 INFO:teuthology.orchestra.run.smithi007.stdout:Size : 8264254 2023-11-27T09:10:00.178 INFO:teuthology.orchestra.run.smithi007.stdout:License : GPLv3+ 2023-11-27T09:10:00.179 INFO:teuthology.orchestra.run.smithi007.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-11-27T09:10:00.179 INFO:teuthology.orchestra.run.smithi007.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-11-27T09:10:00.179 INFO:teuthology.orchestra.run.smithi007.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-11-27T09:10:00.179 INFO:teuthology.orchestra.run.smithi007.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-11-27T09:10:00.179 INFO:teuthology.orchestra.run.smithi007.stdout:Packager : builder@centos.org 2023-11-27T09:10:00.179 INFO:teuthology.orchestra.run.smithi007.stdout:Vendor : CentOS 2023-11-27T09:10:00.179 INFO:teuthology.orchestra.run.smithi007.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-27T09:10:00.179 INFO:teuthology.orchestra.run.smithi007.stdout:Summary : Support tools for GRUB. 2023-11-27T09:10:00.179 INFO:teuthology.orchestra.run.smithi007.stdout:Description : 2023-11-27T09:10:00.179 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:10:00.179 INFO:teuthology.orchestra.run.smithi007.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-27T09:10:00.179 INFO:teuthology.orchestra.run.smithi007.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-27T09:10:00.179 INFO:teuthology.orchestra.run.smithi007.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-27T09:10:00.179 INFO:teuthology.orchestra.run.smithi007.stdout:hardware devices. 2023-11-27T09:10:00.180 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:10:00.180 INFO:teuthology.orchestra.run.smithi007.stdout:This subpackage provides tools for support of all platforms. 2023-11-27T09:10:00.180 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-27T09:10:00.180 INFO:teuthology.task.kernel:Updating grub on smithi007 to boot 5.14.0-386.el9.x86_64 2023-11-27T09:10:00.181 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-27T09:10:00.821 INFO:teuthology.orchestra.run.smithi007.stderr:Generating grub configuration file ... 2023-11-27T09:10:02.249 INFO:teuthology.orchestra.run.smithi007.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-11-27T09:10:02.280 INFO:teuthology.orchestra.run.smithi007.stderr:done 2023-11-27T09:10:02.282 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2023-11-27T09:10:02.298 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.WWOB53P4c5 2023-11-27T09:10:02.298 DEBUG:teuthology.orchestra.run.smithi007:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WWOB53P4c5 2023-11-27T09:10:02.369 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.WWOB53P4c5 2023-11-27T09:10:02.512 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.WWOB53P4c5 is 6KB 2023-11-27T09:10:02.559 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.WWOB53P4c5 2023-11-27T09:10:02.574 DEBUG:teuthology.orchestra.run.smithi007:> sudo /bin/ls /boot/loader/entries || true 2023-11-27T09:10:02.646 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-11-27T09:10:02.646 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-11-27T09:10:02.646 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-11-27T09:10:02.646 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-386.el9.x86_64.conf 2023-11-27T09:10:02.647 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-386.el9.x86_64 2023-11-27T09:10:02.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo shutdown -r now 2023-11-27T09:10:32.831 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-11-27T09:10:32.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-11-27T09:10:32.832 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-27T09:11:32.857 DEBUG:teuthology.orchestra.remote:timed out 2023-11-27T09:11:41.862 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-11-27T09:11:41.863 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-27T09:11:42.141 DEBUG:teuthology.orchestra.run.smithi007:> true 2023-11-27T09:11:42.519 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-11-27T09:11:42.519 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-386.el9.x86_64"... 2023-11-27T09:11:42.519 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2023-11-27T09:11:42.534 INFO:teuthology.orchestra.run.smithi007.stdout:5.14.0-386.el9.x86_64 2023-11-27T09:11:42.534 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-386.el9.x86_64 vs 5.14.0-386.el9.x86_64 2023-11-27T09:11:42.534 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-27T09:11:42.534 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-27T09:11:43.535 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-11-27T09:11:43.535 DEBUG:teuthology.orchestra.run.smithi007:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-27T09:11:43.606 INFO:teuthology.orchestra.run.smithi007.stdout:ttyS1 2023-11-27T09:11:43.649 DEBUG:teuthology.parallel:result is None 2023-11-27T09:11:43.649 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-27T09:11:43.657 INFO:teuthology.task.internal:Creating test directory... 2023-11-27T09:11:43.658 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-27T09:11:43.683 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-27T09:11:43.689 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-27T09:11:43.749 INFO:teuthology.task.internal:Creating archive directory... 2023-11-27T09:11:43.750 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-27T09:11:43.779 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-27T09:11:43.786 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-27T09:11:43.786 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-27T09:11:43.846 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-27T09:11:43.863 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-27T09:11:43.865 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-27T09:11:43.872 INFO:teuthology.task.internal:Configuring sudo... 2023-11-27T09:11:43.872 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-27T09:11:43.932 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-27T09:11:43.940 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-27T09:11:43.941 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-27T09:11:43.984 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-27T09:11:44.060 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-27T09:11:44.108 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-27T09:11:44.187 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-27T09:11:44.230 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:11:44.230 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-27T09:11:44.297 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2023-11-27T09:11:44.394 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-27T09:11:44.835 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-27T09:11:44.843 INFO:teuthology.task.internal:Starting timer... 2023-11-27T09:11:44.843 INFO:teuthology.run_tasks:Running task pcp... 2023-11-27T09:11:44.852 INFO:teuthology.run_tasks:Running task selinux... 2023-11-27T09:11:44.861 DEBUG:teuthology.orchestra.run.smithi007:> sudo service auditd rotate 2023-11-27T09:11:44.953 INFO:teuthology.orchestra.run.smithi007.stdout:Rotating logs: 2023-11-27T09:11:44.955 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-27T09:11:44.956 DEBUG:teuthology.orchestra.run.smithi007:> /usr/sbin/getenforce 2023-11-27T09:11:44.992 INFO:teuthology.orchestra.run.smithi007.stdout:Permissive 2023-11-27T09:11:44.992 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi007.front.sepia.ceph.com': 'permissive'} 2023-11-27T09:11:44.992 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-27T09:11:45.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-27T09:11:45.084 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-27T09:11:45.085 DEBUG:teuthology.orchestra.run.smithi007:> sudo /usr/sbin/setenforce permissive 2023-11-27T09:11:45.395 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-27T09:11:45.404 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-27T09:11:45.404 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-27T09:11:45.419 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-27T09:11:45.420 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-27T09:17:52.907 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com')] 2023-11-27T09:17:52.908 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-11-27T09:17:52.909 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-27T09:17:53.009 DEBUG:teuthology.orchestra.run.smithi007:> true 2023-11-27T09:17:53.078 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2023-11-27T09:17:53.079 INFO:teuthology.run_tasks:Running task clock... 2023-11-27T09:17:53.090 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-27T09:17:53.090 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-27T09:17:53.091 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-27T09:17:53.156 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-27T09:17:53.177 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-27T09:17:53.215 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: ntpd: command not found 2023-11-27T09:17:53.232 INFO:teuthology.orchestra.run.smithi007.stdout:506 Cannot talk to daemon 2023-11-27T09:17:53.252 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-27T09:17:53.272 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-27T09:17:53.321 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2023-11-27T09:17:53.322 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-27T09:17:53.322 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2023-11-27T09:17:53.322 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-27T09:17:53.322 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-27T09:17:53.322 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-27T09:17:53.322 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-27T09:17:53.323 INFO:teuthology.run_tasks:Running task install... 2023-11-27T09:17:53.333 DEBUG:teuthology.task.install:project ceph 2023-11-27T09:17:53.333 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3'}} 2023-11-27T09:17:53.334 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3'} 2023-11-27T09:17:53.334 INFO:teuthology.task.install:Using flavor: crimson 2023-11-27T09:17:53.341 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-27T09:17:53.341 INFO:teuthology.task.install:extra packages: [] 2023-11-27T09:17:53.341 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3', 'tag': None, 'wait_for_package': False} 2023-11-27T09:17:53.342 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3 2023-11-27T09:17:53.553 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yingxin-crimson-make-crosscore-send-ordered/705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3/centos/9/flavors/crimson/ 2023-11-27T09:17:53.553 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7476.g705aae7f 2023-11-27T09:17:53.725 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yingxin-crimson-make-crosscore-send-ordered/705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yingxin-crimson-make-crosscore-send-ordered/705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yingxin-crimson-make-crosscore-send-ordered/705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-27T09:17:53.726 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:17:53.726 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-27T09:17:53.759 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-27T09:17:53.759 DEBUG:teuthology.orchestra.run.smithi007:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-27T09:17:53.831 DEBUG:teuthology.orchestra.run.smithi007:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-27T09:17:53.920 DEBUG:teuthology.orchestra.run.smithi007:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-27T09:17:53.950 INFO:teuthology.orchestra.run.smithi007.stdout:check_obsoletes = 1 2023-11-27T09:17:53.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum clean all 2023-11-27T09:17:54.211 INFO:teuthology.orchestra.run.smithi007.stdout:73 files removed 2023-11-27T09:17:54.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-radosgw 2023-11-27T09:17:55.062 INFO:teuthology.orchestra.run.smithi007.stdout:ceph packages for x86_64 167 kB/s | 82 kB 00:00 2023-11-27T09:17:55.496 INFO:teuthology.orchestra.run.smithi007.stdout:ceph noarch packages 25 kB/s | 9.5 kB 00:00 2023-11-27T09:17:55.888 INFO:teuthology.orchestra.run.smithi007.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2023-11-27T09:17:56.290 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - AppStream 18 MB/s | 6.4 MB 00:00 2023-11-27T09:17:57.657 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2023-11-27T09:18:00.473 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - BaseOS 3.2 MB/s | 7.9 MB 00:02 2023-11-27T09:18:02.440 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - AppStream 23 MB/s | 18 MB 00:00 2023-11-27T09:18:07.586 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 5.8 MB 00:01 2023-11-27T09:18:09.300 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2023-11-27T09:18:10.694 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 16 MB/s | 20 MB 00:01 2023-11-27T09:18:15.710 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2023-11-27T09:18:18.546 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:18:18.550 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:18:18.550 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-11-27T09:18:18.550 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:18:18.550 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-27T09:18:18.550 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 8.4 M 2023-11-27T09:18:18.550 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrading: 2023-11-27T09:18:18.551 INFO:teuthology.orchestra.run.smithi007.stdout: librados2 x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 3.1 M 2023-11-27T09:18:18.551 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1 x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 3.0 M 2023-11-27T09:18:18.551 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-11-27T09:18:18.552 INFO:teuthology.orchestra.run.smithi007.stdout: boost-program-options 2023-11-27T09:18:18.552 INFO:teuthology.orchestra.run.smithi007.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-11-27T09:18:18.552 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 6.6 M 2023-11-27T09:18:18.553 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 17 M 2023-11-27T09:18:18.553 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 24 k 2023-11-27T09:18:18.553 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-11-27T09:18:18.553 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-11-27T09:18:18.553 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-11-27T09:18:18.554 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2 x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 729 k 2023-11-27T09:18:18.554 INFO:teuthology.orchestra.run.smithi007.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-11-27T09:18:18.554 INFO:teuthology.orchestra.run.smithi007.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-11-27T09:18:18.554 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-11-27T09:18:18.554 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1 x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 470 k 2023-11-27T09:18:18.554 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-11-27T09:18:18.555 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2 x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 4.5 M 2023-11-27T09:18:18.555 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-11-27T09:18:18.555 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-11-27T09:18:18.555 INFO:teuthology.orchestra.run.smithi007.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-11-27T09:18:18.555 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse 2023-11-27T09:18:18.555 INFO:teuthology.orchestra.run.smithi007.stdout: x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 45 k 2023-11-27T09:18:18.555 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 129 k 2023-11-27T09:18:18.555 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 161 k 2023-11-27T09:18:18.555 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-11-27T09:18:18.555 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 321 k 2023-11-27T09:18:18.556 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 298 k 2023-11-27T09:18:18.556 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 99 k 2023-11-27T09:18:18.556 INFO:teuthology.orchestra.run.smithi007.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-11-27T09:18:18.556 INFO:teuthology.orchestra.run.smithi007.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-11-27T09:18:18.556 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:18:18.556 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-27T09:18:18.556 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:18:18.556 INFO:teuthology.orchestra.run.smithi007.stdout:Install 27 Packages 2023-11-27T09:18:18.556 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrade 2 Packages 2023-11-27T09:18:18.556 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:18:18.557 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 54 M 2023-11-27T09:18:18.557 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-27T09:18:20.477 INFO:teuthology.orchestra.run.smithi007.stdout:(1/29): ceph-radosgw-18.0.0-7476.g705aae7f.el9. 5.4 MB/s | 8.4 MB 00:01 2023-11-27T09:18:20.511 INFO:teuthology.orchestra.run.smithi007.stdout:(2/29): ceph-selinux-18.0.0-7476.g705aae7f.el9. 730 kB/s | 24 kB 00:00 2023-11-27T09:18:20.603 INFO:teuthology.orchestra.run.smithi007.stdout:(3/29): libcephfs2-18.0.0-7476.g705aae7f.el9.x8 7.8 MB/s | 729 kB 00:00 2023-11-27T09:18:20.686 INFO:teuthology.orchestra.run.smithi007.stdout:(4/29): libradosstriper1-18.0.0-7476.g705aae7f. 5.5 MB/s | 470 kB 00:00 2023-11-27T09:18:20.912 INFO:teuthology.orchestra.run.smithi007.stdout:(5/29): ceph-base-18.0.0-7476.g705aae7f.el9.x86 3.3 MB/s | 6.6 MB 00:01 2023-11-27T09:18:20.962 INFO:teuthology.orchestra.run.smithi007.stdout:(6/29): python3-ceph-argparse-18.0.0-7476.g705a 905 kB/s | 45 kB 00:00 2023-11-27T09:18:21.029 INFO:teuthology.orchestra.run.smithi007.stdout:(7/29): python3-ceph-common-18.0.0-7476.g705aae 1.9 MB/s | 129 kB 00:00 2023-11-27T09:18:21.104 INFO:teuthology.orchestra.run.smithi007.stdout:(8/29): python3-cephfs-18.0.0-7476.g705aae7f.el 2.1 MB/s | 161 kB 00:00 2023-11-27T09:18:21.190 INFO:teuthology.orchestra.run.smithi007.stdout:(9/29): librgw2-18.0.0-7476.g705aae7f.el9.x86_6 9.0 MB/s | 4.5 MB 00:00 2023-11-27T09:18:21.213 INFO:teuthology.orchestra.run.smithi007.stdout:(10/29): python3-rados-18.0.0-7476.g705aae7f.el 2.9 MB/s | 321 kB 00:00 2023-11-27T09:18:21.263 INFO:teuthology.orchestra.run.smithi007.stdout:(11/29): python3-rbd-18.0.0-7476.g705aae7f.el9. 3.9 MB/s | 298 kB 00:00 2023-11-27T09:18:21.289 INFO:teuthology.orchestra.run.smithi007.stdout:(12/29): python3-rgw-18.0.0-7476.g705aae7f.el9. 1.3 MB/s | 99 kB 00:00 2023-11-27T09:18:21.405 INFO:teuthology.orchestra.run.smithi007.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 747 kB/s | 106 kB 00:00 2023-11-27T09:18:21.439 INFO:teuthology.orchestra.run.smithi007.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.3 MB/s | 159 kB 00:00 2023-11-27T09:18:21.465 INFO:teuthology.orchestra.run.smithi007.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 751 kB/s | 44 kB 00:00 2023-11-27T09:18:21.506 INFO:teuthology.orchestra.run.smithi007.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2023-11-27T09:18:21.532 INFO:teuthology.orchestra.run.smithi007.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2023-11-27T09:18:21.557 INFO:teuthology.orchestra.run.smithi007.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 827 kB/s | 41 kB 00:00 2023-11-27T09:18:22.140 INFO:teuthology.orchestra.run.smithi007.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 508 kB/s | 309 kB 00:00 2023-11-27T09:18:22.232 INFO:teuthology.orchestra.run.smithi007.stdout:(20/29): libarrow-doc-9.0.0-9.el9.noarch.rpm 266 kB/s | 24 kB 00:00 2023-11-27T09:18:22.333 INFO:teuthology.orchestra.run.smithi007.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 486 kB/s | 49 kB 00:00 2023-11-27T09:18:22.461 INFO:teuthology.orchestra.run.smithi007.stdout:(22/29): libarrow-9.0.0-9.el9.x86_64.rpm 4.9 MB/s | 4.4 MB 00:00 2023-11-27T09:18:22.486 INFO:teuthology.orchestra.run.smithi007.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 442 kB/s | 67 kB 00:00 2023-11-27T09:18:22.569 INFO:teuthology.orchestra.run.smithi007.stdout:(24/29): parquet-libs-9.0.0-9.el9.x86_64.rpm 7.6 MB/s | 837 kB 00:00 2023-11-27T09:18:22.595 INFO:teuthology.orchestra.run.smithi007.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2023-11-27T09:18:22.695 INFO:teuthology.orchestra.run.smithi007.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2023-11-27T09:18:23.062 INFO:teuthology.orchestra.run.smithi007.stdout:(27/29): librbd1-18.0.0-7476.g705aae7f.el9.x86_ 8.1 MB/s | 3.0 MB 00:00 2023-11-27T09:18:23.154 INFO:teuthology.orchestra.run.smithi007.stdout:(28/29): librados2-18.0.0-7476.g705aae7f.el9.x8 5.6 MB/s | 3.1 MB 00:00 2023-11-27T09:18:23.454 INFO:teuthology.orchestra.run.smithi007.stdout:(29/29): ceph-common-18.0.0-7476.g705aae7f.el9. 3.8 MB/s | 17 MB 00:04 2023-11-27T09:18:23.462 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-27T09:18:23.462 INFO:teuthology.orchestra.run.smithi007.stdout:Total 11 MB/s | 54 MB 00:04 2023-11-27T09:18:23.462 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-27T09:18:23.503 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-27T09:18:23.503 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-27T09:18:23.978 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-27T09:18:23.979 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-27T09:18:25.384 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-27T09:18:25.803 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-11-27T09:18:25.831 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librados2-2:18.0.0-7476.g705aae7f.el9.x86_64 2/31 2023-11-27T09:18:25.924 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-2:18.0.0-7476.g705aae7f.el9.x86_64 2/31 2023-11-27T09:18:26.014 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rados-2:18.0.0-7476.g705aae7f.el9.x86_64 3/31 2023-11-27T09:18:26.041 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs2-2:18.0.0-7476.g705aae7f.el9.x86_64 4/31 2023-11-27T09:18:26.121 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs2-2:18.0.0-7476.g705aae7f.el9.x86_64 4/31 2023-11-27T09:18:26.153 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/31 2023-11-27T09:18:26.178 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/31 2023-11-27T09:18:26.209 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/31 2023-11-27T09:18:26.251 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-argparse-2:18.0.0-7476.g705aae7f.el9. 8/31 2023-11-27T09:18:26.312 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cephfs-2:18.0.0-7476.g705aae7f.el9.x86_64 9/31 2023-11-27T09:18:26.336 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libradosstriper1-2:18.0.0-7476.g705aae7f.el9.x86_6 10/31 2023-11-27T09:18:26.424 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7476.g705aae7f.el9.x86_6 10/31 2023-11-27T09:18:26.459 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/31 2023-11-27T09:18:26.570 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-11-27T09:18:26.602 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/31 2023-11-27T09:18:26.643 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-11-27T09:18:26.683 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-11-27T09:18:26.712 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/31 2023-11-27T09:18:26.745 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-11-27T09:18:27.075 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-11-27T09:18:27.108 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librbd1-2:18.0.0-7476.g705aae7f.el9.x86_64 19/31 2023-11-27T09:18:27.204 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-2:18.0.0-7476.g705aae7f.el9.x86_64 19/31 2023-11-27T09:18:27.232 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rbd-2:18.0.0-7476.g705aae7f.el9.x86_64 20/31 2023-11-27T09:18:27.268 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-11-27T09:18:27.723 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 22/31 2023-11-27T09:18:27.747 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librgw2-2:18.0.0-7476.g705aae7f.el9.x86_64 23/31 2023-11-27T09:18:27.824 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librgw2-2:18.0.0-7476.g705aae7f.el9.x86_64 23/31 2023-11-27T09:18:27.896 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rgw-2:18.0.0-7476.g705aae7f.el9.x86_64 24/31 2023-11-27T09:18:27.924 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-common-2:18.0.0-7476.g705aae7f.el9.x8 25/31 2023-11-27T09:18:30.109 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:18.0.0-7476.g705aae7f.el9.x86_64 26/31 2023-11-27T09:18:30.146 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-common-2:18.0.0-7476.g705aae7f.el9.x86_64 26/31 2023-11-27T09:18:30.922 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:18.0.0-7476.g705aae7f.el9.x86_64 26/31 2023-11-27T09:18:30.958 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-base-2:18.0.0-7476.g705aae7f.el9.x86_64 27/31 2023-11-27T09:18:31.048 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-base-2:18.0.0-7476.g705aae7f.el9.x86_64 27/31 2023-11-27T09:18:31.048 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-11-27T09:18:31.048 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-11-27T09:18:31.048 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:18:31.075 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-selinux-2:18.0.0-7476.g705aae7f.el9.x86_64 28/31 2023-11-27T09:18:41.289 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7476.g705aae7f.el9.x86_64 28/31 2023-11-27T09:18:41.290 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /sys 2023-11-27T09:18:41.290 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /proc 2023-11-27T09:18:41.290 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /mnt 2023-11-27T09:18:41.290 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /var/tmp 2023-11-27T09:18:41.290 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /home 2023-11-27T09:18:41.290 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /root 2023-11-27T09:18:41.290 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /tmp 2023-11-27T09:18:41.290 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:18:42.158 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-radosgw-2:18.0.0-7476.g705aae7f.el9.x86_64 29/31 2023-11-27T09:18:42.184 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7476.g705aae7f.el9.x86_64 29/31 2023-11-27T09:18:42.184 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-27T09:18:42.184 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-11-27T09:18:42.184 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-27T09:18:42.185 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-27T09:18:42.185 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:18:42.186 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-11-27T09:18:42.268 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-11-27T09:18:42.268 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-11-27T09:18:43.337 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-11-27T09:18:43.337 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-base-2:18.0.0-7476.g705aae7f.el9.x86_64 1/31 2023-11-27T09:18:43.337 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-common-2:18.0.0-7476.g705aae7f.el9.x86_64 2/31 2023-11-27T09:18:43.338 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-radosgw-2:18.0.0-7476.g705aae7f.el9.x86_64 3/31 2023-11-27T09:18:43.338 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-selinux-2:18.0.0-7476.g705aae7f.el9.x86_64 4/31 2023-11-27T09:18:43.338 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs2-2:18.0.0-7476.g705aae7f.el9.x86_64 5/31 2023-11-27T09:18:43.338 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libradosstriper1-2:18.0.0-7476.g705aae7f.el9.x86_6 6/31 2023-11-27T09:18:43.338 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librgw2-2:18.0.0-7476.g705aae7f.el9.x86_64 7/31 2023-11-27T09:18:43.338 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7476.g705aae7f.el9. 8/31 2023-11-27T09:18:43.338 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-common-2:18.0.0-7476.g705aae7f.el9.x8 9/31 2023-11-27T09:18:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cephfs-2:18.0.0-7476.g705aae7f.el9.x86_64 10/31 2023-11-27T09:18:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rados-2:18.0.0-7476.g705aae7f.el9.x86_64 11/31 2023-11-27T09:18:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rbd-2:18.0.0-7476.g705aae7f.el9.x86_64 12/31 2023-11-27T09:18:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rgw-2:18.0.0-7476.g705aae7f.el9.x86_64 13/31 2023-11-27T09:18:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-11-27T09:18:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-11-27T09:18:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-11-27T09:18:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-11-27T09:18:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-11-27T09:18:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-11-27T09:18:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-11-27T09:18:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 21/31 2023-11-27T09:18:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 22/31 2023-11-27T09:18:43.339 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-11-27T09:18:43.340 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-11-27T09:18:43.340 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 25/31 2023-11-27T09:18:43.340 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-11-27T09:18:43.340 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-11-27T09:18:43.340 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-2:18.0.0-7476.g705aae7f.el9.x86_64 28/31 2023-11-27T09:18:43.340 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-11-27T09:18:43.340 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-2:18.0.0-7476.g705aae7f.el9.x86_64 30/31 2023-11-27T09:18:44.272 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-11-27T09:18:44.272 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:18:44.272 INFO:teuthology.orchestra.run.smithi007.stdout:Upgraded: 2023-11-27T09:18:44.272 INFO:teuthology.orchestra.run.smithi007.stdout: librados2-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:18:44.272 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:18:44.272 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-27T09:18:44.272 INFO:teuthology.orchestra.run.smithi007.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-11-27T09:18:44.272 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:18:44.272 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:18:44.272 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:18:44.273 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:18:44.273 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-11-27T09:18:44.273 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-9.0.0-9.el9.x86_64 2023-11-27T09:18:44.273 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-11-27T09:18:44.273 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:18:44.273 INFO:teuthology.orchestra.run.smithi007.stdout: liboath-2.6.7-2.el9.x86_64 2023-11-27T09:18:44.273 INFO:teuthology.orchestra.run.smithi007.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-11-27T09:18:44.273 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-11-27T09:18:44.273 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:18:44.273 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-11-27T09:18:44.273 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:18:44.273 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind-1.6.2-1.el9.x86_64 2023-11-27T09:18:44.273 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-11-27T09:18:44.274 INFO:teuthology.orchestra.run.smithi007.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-11-27T09:18:44.274 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:18:44.274 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:18:44.274 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:18:44.274 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-11-27T09:18:44.274 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:18:44.274 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:18:44.274 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:18:44.274 INFO:teuthology.orchestra.run.smithi007.stdout: re2-1:20211101-3.el9.x86_64 2023-11-27T09:18:44.274 INFO:teuthology.orchestra.run.smithi007.stdout: thrift-0.15.0-2.el9.x86_64 2023-11-27T09:18:44.274 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:18:44.274 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:18:44.603 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-test 2023-11-27T09:18:45.222 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:30 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:18:45.378 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:18:45.379 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:18:45.379 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-11-27T09:18:45.379 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:18:45.379 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-27T09:18:45.379 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 117 M 2023-11-27T09:18:45.379 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-11-27T09:18:45.379 INFO:teuthology.orchestra.run.smithi007.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2023-11-27T09:18:45.379 INFO:teuthology.orchestra.run.smithi007.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-11-27T09:18:45.379 INFO:teuthology.orchestra.run.smithi007.stdout: libasan x86_64 11.4.1-2.3.el9 CentOS-AppStream 418 k 2023-11-27T09:18:45.379 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 158 k 2023-11-27T09:18:45.379 INFO:teuthology.orchestra.run.smithi007.stdout: libubsan x86_64 11.4.1-2.3.el9 CentOS-AppStream 186 k 2023-11-27T09:18:45.380 INFO:teuthology.orchestra.run.smithi007.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-11-27T09:18:45.380 INFO:teuthology.orchestra.run.smithi007.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-11-27T09:18:45.380 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-11-27T09:18:45.380 INFO:teuthology.orchestra.run.smithi007.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2023-11-27T09:18:45.380 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:18:45.380 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-27T09:18:45.380 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:18:45.380 INFO:teuthology.orchestra.run.smithi007.stdout:Install 10 Packages 2023-11-27T09:18:45.380 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:18:45.381 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 120 M 2023-11-27T09:18:45.381 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 1.1 G 2023-11-27T09:18:45.381 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-27T09:18:45.581 INFO:teuthology.orchestra.run.smithi007.stdout:(1/10): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 186 kB 00:00 2023-11-27T09:18:45.631 INFO:teuthology.orchestra.run.smithi007.stdout:(2/10): libasan-11.4.1-2.3.el9.x86_64.rpm 8.5 MB/s | 418 kB 00:00 2023-11-27T09:18:45.664 INFO:teuthology.orchestra.run.smithi007.stdout:(3/10): libubsan-11.4.1-2.3.el9.x86_64.rpm 5.5 MB/s | 186 kB 00:00 2023-11-27T09:18:45.698 INFO:teuthology.orchestra.run.smithi007.stdout:(4/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 6.3 MB/s | 217 kB 00:00 2023-11-27T09:18:45.740 INFO:teuthology.orchestra.run.smithi007.stdout:(5/10): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-11-27T09:18:45.774 INFO:teuthology.orchestra.run.smithi007.stdout:(6/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-11-27T09:18:45.949 INFO:teuthology.orchestra.run.smithi007.stdout:(7/10): libcephsqlite-18.0.0-7476.g705aae7f.el9 307 kB/s | 158 kB 00:00 2023-11-27T09:18:46.574 INFO:teuthology.orchestra.run.smithi007.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 195 kB/s | 122 kB 00:00 2023-11-27T09:18:47.799 INFO:teuthology.orchestra.run.smithi007.stdout:(9/10): cryptopp-8.6.0-1.el9.x86_64.rpm 635 kB/s | 1.3 MB 00:02 2023-11-27T09:19:23.703 INFO:teuthology.orchestra.run.smithi007.stdout:(10/10): ceph-test-18.0.0-7476.g705aae7f.el9.x8 3.1 MB/s | 117 MB 00:38 2023-11-27T09:19:23.706 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-27T09:19:23.706 INFO:teuthology.orchestra.run.smithi007.stdout:Total 3.1 MB/s | 120 MB 00:38 2023-11-27T09:19:23.707 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-27T09:19:23.727 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-27T09:19:23.728 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-27T09:19:24.582 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-27T09:19:24.582 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-27T09:19:25.791 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-27T09:19:25.882 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2023-11-27T09:19:25.911 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2023-11-27T09:19:25.955 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2023-11-27T09:19:25.989 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2023-11-27T09:19:26.020 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-11-27T09:19:26.052 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : jq-1.6-16.el9.x86_64 6/10 2023-11-27T09:19:26.087 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libubsan-11.4.1-2.3.el9.x86_64 7/10 2023-11-27T09:19:26.131 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libasan-11.4.1-2.3.el9.x86_64 8/10 2023-11-27T09:19:26.158 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephsqlite-2:18.0.0-7476.g705aae7f.el9.x86_64 9/10 2023-11-27T09:19:43.419 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7476.g705aae7f.el9.x86_64 9/10 2023-11-27T09:19:43.493 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-test-2:18.0.0-7476.g705aae7f.el9.x86_64 10/10 2023-11-27T09:19:51.899 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-test-2:18.0.0-7476.g705aae7f.el9.x86_64 10/10 2023-11-27T09:19:51.899 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-test-2:18.0.0-7476.g705aae7f.el9.x86_64 1/10 2023-11-27T09:19:51.899 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephsqlite-2:18.0.0-7476.g705aae7f.el9.x86_64 2/10 2023-11-27T09:19:51.899 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : jq-1.6-16.el9.x86_64 3/10 2023-11-27T09:19:51.899 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libasan-11.4.1-2.3.el9.x86_64 4/10 2023-11-27T09:19:51.899 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libubsan-11.4.1-2.3.el9.x86_64 5/10 2023-11-27T09:19:51.899 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/10 2023-11-27T09:19:51.899 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 7/10 2023-11-27T09:19:51.899 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/10 2023-11-27T09:19:51.899 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2023-11-27T09:19:52.302 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2023-11-27T09:19:52.302 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:19:52.302 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-27T09:19:52.302 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:19:52.303 INFO:teuthology.orchestra.run.smithi007.stdout: cryptopp-8.6.0-1.el9.x86_64 2023-11-27T09:19:52.303 INFO:teuthology.orchestra.run.smithi007.stdout: jq-1.6-16.el9.x86_64 2023-11-27T09:19:52.303 INFO:teuthology.orchestra.run.smithi007.stdout: libasan-11.4.1-2.3.el9.x86_64 2023-11-27T09:19:52.303 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:19:52.303 INFO:teuthology.orchestra.run.smithi007.stdout: libubsan-11.4.1-2.3.el9.x86_64 2023-11-27T09:19:52.303 INFO:teuthology.orchestra.run.smithi007.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-11-27T09:19:52.303 INFO:teuthology.orchestra.run.smithi007.stdout: socat-1.7.4.1-5.el9.x86_64 2023-11-27T09:19:52.303 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-11-27T09:19:52.303 INFO:teuthology.orchestra.run.smithi007.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2023-11-27T09:19:52.303 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:19:52.303 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:19:52.578 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph 2023-11-27T09:19:53.179 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:01:38 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:19:53.338 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:19:53.340 INFO:teuthology.orchestra.run.smithi007.stdout:========================================================================================= 2023-11-27T09:19:53.340 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-11-27T09:19:53.340 INFO:teuthology.orchestra.run.smithi007.stdout:========================================================================================= 2023-11-27T09:19:53.340 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-27T09:19:53.340 INFO:teuthology.orchestra.run.smithi007.stdout: ceph x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 6.5 k 2023-11-27T09:19:53.340 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-11-27T09:19:53.341 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 2.0 M 2023-11-27T09:19:53.341 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 1.5 M 2023-11-27T09:19:53.341 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7476.g705aae7f.el9 ceph-noarch 243 k 2023-11-27T09:19:53.341 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 8.9 M 2023-11-27T09:19:53.341 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 117 M 2023-11-27T09:19:53.341 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-11-27T09:19:53.341 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-11-27T09:19:53.341 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-11-27T09:19:53.341 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-11-27T09:19:53.341 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-11-27T09:19:53.341 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-11-27T09:19:53.341 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-11-27T09:19:53.342 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-11-27T09:19:53.342 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-11-27T09:19:53.342 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-11-27T09:19:53.342 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-11-27T09:19:53.342 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-11-27T09:19:53.342 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-11-27T09:19:53.342 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-11-27T09:19:53.342 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-11-27T09:19:53.342 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-11-27T09:19:53.342 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-11-27T09:19:53.342 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-11-27T09:19:53.342 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-11-27T09:19:53.342 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-11-27T09:19:53.343 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-11-27T09:19:53.343 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-11-27T09:19:53.343 INFO:teuthology.orchestra.run.smithi007.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-11-27T09:19:53.343 INFO:teuthology.orchestra.run.smithi007.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-11-27T09:19:53.343 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-11-27T09:19:53.343 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-11-27T09:19:53.343 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-11-27T09:19:53.343 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:19:53.343 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-27T09:19:53.343 INFO:teuthology.orchestra.run.smithi007.stdout:========================================================================================= 2023-11-27T09:19:53.343 INFO:teuthology.orchestra.run.smithi007.stdout:Install 33 Packages 2023-11-27T09:19:53.343 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:19:53.344 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 134 M 2023-11-27T09:19:53.344 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 1.4 G 2023-11-27T09:19:53.344 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-27T09:19:54.109 INFO:teuthology.orchestra.run.smithi007.stdout:(1/33): ceph-18.0.0-7476.g705aae7f.el9.x86_64.r 16 kB/s | 6.5 kB 00:00 2023-11-27T09:19:55.784 INFO:teuthology.orchestra.run.smithi007.stdout:(2/33): ceph-mds-18.0.0-7476.g705aae7f.el9.x86_ 986 kB/s | 2.0 MB 00:02 2023-11-27T09:19:55.826 INFO:teuthology.orchestra.run.smithi007.stdout:(3/33): ceph-mgr-18.0.0-7476.g705aae7f.el9.x86_ 702 kB/s | 1.5 MB 00:02 2023-11-27T09:19:56.026 INFO:teuthology.orchestra.run.smithi007.stdout:(4/33): ceph-mgr-modules-core-18.0.0-7476.g705a 1.2 MB/s | 243 kB 00:00 2023-11-27T09:19:56.126 INFO:teuthology.orchestra.run.smithi007.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 144 kB 00:00 2023-11-27T09:19:56.160 INFO:teuthology.orchestra.run.smithi007.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 950 kB/s | 32 kB 00:00 2023-11-27T09:19:56.194 INFO:teuthology.orchestra.run.smithi007.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2023-11-27T09:19:56.227 INFO:teuthology.orchestra.run.smithi007.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-11-27T09:19:56.269 INFO:teuthology.orchestra.run.smithi007.stdout:(9/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2023-11-27T09:19:56.340 INFO:teuthology.orchestra.run.smithi007.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 16 MB/s | 1.1 MB 00:00 2023-11-27T09:19:56.373 INFO:teuthology.orchestra.run.smithi007.stdout:(11/33): python3-ply-3.11-14.el9.noarch.rpm 3.1 MB/s | 103 kB 00:00 2023-11-27T09:19:56.407 INFO:teuthology.orchestra.run.smithi007.stdout:(12/33): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2023-11-27T09:19:56.441 INFO:teuthology.orchestra.run.smithi007.stdout:(13/33): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-11-27T09:19:56.475 INFO:teuthology.orchestra.run.smithi007.stdout:(14/33): python3-urllib3-1.26.5-4.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2023-11-27T09:19:56.825 INFO:teuthology.orchestra.run.smithi007.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 124 kB/s | 43 kB 00:00 2023-11-27T09:19:57.108 INFO:teuthology.orchestra.run.smithi007.stdout:(16/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 606 kB/s | 172 kB 00:00 2023-11-27T09:19:57.317 INFO:teuthology.orchestra.run.smithi007.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2023-11-27T09:19:57.425 INFO:teuthology.orchestra.run.smithi007.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 99 kB/s | 11 kB 00:00 2023-11-27T09:19:57.576 INFO:teuthology.orchestra.run.smithi007.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 118 kB/s | 18 kB 00:00 2023-11-27T09:19:57.701 INFO:teuthology.orchestra.run.smithi007.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 185 kB/s | 23 kB 00:00 2023-11-27T09:19:57.827 INFO:teuthology.orchestra.run.smithi007.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 155 kB/s | 19 kB 00:00 2023-11-27T09:19:57.952 INFO:teuthology.orchestra.run.smithi007.stdout:(22/33): python3-jaraco-text-3.2.0-6.el9.noarch 157 kB/s | 20 kB 00:00 2023-11-27T09:19:58.077 INFO:teuthology.orchestra.run.smithi007.stdout:(23/33): python3-logutils-0.3.5-21.el9.noarch.r 371 kB/s | 46 kB 00:00 2023-11-27T09:19:58.202 INFO:teuthology.orchestra.run.smithi007.stdout:(24/33): python3-more-itertools-8.12.0-2.el9.no 632 kB/s | 79 kB 00:00 2023-11-27T09:19:58.428 INFO:teuthology.orchestra.run.smithi007.stdout:(25/33): ceph-mon-18.0.0-7476.g705aae7f.el9.x86 2.1 MB/s | 8.9 MB 00:04 2023-11-27T09:19:58.453 INFO:teuthology.orchestra.run.smithi007.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.1 MB/s | 272 kB 00:00 2023-11-27T09:19:58.587 INFO:teuthology.orchestra.run.smithi007.stdout:(27/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 675 kB/s | 90 kB 00:00 2023-11-27T09:19:58.720 INFO:teuthology.orchestra.run.smithi007.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 56 kB/s | 16 kB 00:00 2023-11-27T09:19:58.737 INFO:teuthology.orchestra.run.smithi007.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 238 kB/s | 36 kB 00:00 2023-11-27T09:19:58.963 INFO:teuthology.orchestra.run.smithi007.stdout:(30/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.9 MB/s | 427 kB 00:00 2023-11-27T09:19:59.129 INFO:teuthology.orchestra.run.smithi007.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 120 kB/s | 20 kB 00:00 2023-11-27T09:19:59.163 INFO:teuthology.orchestra.run.smithi007.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 519 kB/s | 230 kB 00:00 2023-11-27T09:23:01.646 INFO:teuthology.orchestra.run.smithi007.stdout:(33/33): ceph-osd-18.0.0-7476.g705aae7f.el9.x86 647 kB/s | 117 MB 03:05 2023-11-27T09:23:01.649 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-27T09:23:01.649 INFO:teuthology.orchestra.run.smithi007.stdout:Total 728 kB/s | 134 MB 03:08 2023-11-27T09:23:01.649 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-27T09:23:01.689 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-27T09:23:01.689 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-27T09:23:02.709 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-27T09:23:02.709 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-27T09:23:04.601 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-27T09:23:04.656 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-11-27T09:23:04.697 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-11-27T09:23:04.738 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-11-27T09:23:04.833 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-11-27T09:23:04.910 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-11-27T09:23:04.975 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-11-27T09:23:05.049 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-11-27T09:23:05.109 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 8/33 2023-11-27T09:23:05.147 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/33 2023-11-27T09:23:05.206 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2023-11-27T09:23:05.269 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2023-11-27T09:23:05.468 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2023-11-27T09:23:05.518 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2023-11-27T09:23:05.560 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2023-11-27T09:23:05.609 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2023-11-27T09:23:05.638 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2023-11-27T09:23:05.718 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2023-11-27T09:23:05.748 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-11-27T09:23:05.782 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2023-11-27T09:23:05.822 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2023-11-27T09:23:05.866 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2023-11-27T09:23:05.900 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-11-27T09:23:05.986 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-11-27T09:23:06.050 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-11-27T09:23:06.132 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-11-27T09:23:06.291 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-11-27T09:23:06.422 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-11-27T09:23:06.597 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7476.g705aae7f.el9. 28/33 2023-11-27T09:23:06.628 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-2:18.0.0-7476.g705aae7f.el9.x86_64 29/33 2023-11-27T09:23:06.653 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7476.g705aae7f.el9.x86_64 29/33 2023-11-27T09:23:06.653 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-27T09:23:06.654 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-11-27T09:23:06.654 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-27T09:23:06.654 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-27T09:23:06.654 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:23:26.018 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-osd-2:18.0.0-7476.g705aae7f.el9.x86_64 30/33 2023-11-27T09:23:26.045 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-osd-2:18.0.0-7476.g705aae7f.el9.x86_64 30/33 2023-11-27T09:23:26.045 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-27T09:23:26.045 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-11-27T09:23:26.045 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-27T09:23:26.045 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-27T09:23:26.045 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:23:26.977 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mon-2:18.0.0-7476.g705aae7f.el9.x86_64 31/33 2023-11-27T09:23:27.000 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mon-2:18.0.0-7476.g705aae7f.el9.x86_64 31/33 2023-11-27T09:23:27.000 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-27T09:23:27.001 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-11-27T09:23:27.001 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-27T09:23:27.001 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-27T09:23:27.001 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:23:27.211 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mds-2:18.0.0-7476.g705aae7f.el9.x86_64 32/33 2023-11-27T09:23:27.234 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mds-2:18.0.0-7476.g705aae7f.el9.x86_64 32/33 2023-11-27T09:23:27.234 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-27T09:23:27.235 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-11-27T09:23:27.235 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-27T09:23:27.235 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-27T09:23:27.235 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:23:27.303 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-2:18.0.0-7476.g705aae7f.el9.x86_64 33/33 2023-11-27T09:23:37.244 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-2:18.0.0-7476.g705aae7f.el9.x86_64 33/33 2023-11-27T09:23:37.245 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-2:18.0.0-7476.g705aae7f.el9.x86_64 1/33 2023-11-27T09:23:37.245 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mds-2:18.0.0-7476.g705aae7f.el9.x86_64 2/33 2023-11-27T09:23:37.245 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-2:18.0.0-7476.g705aae7f.el9.x86_64 3/33 2023-11-27T09:23:37.245 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mon-2:18.0.0-7476.g705aae7f.el9.x86_64 4/33 2023-11-27T09:23:37.245 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-osd-2:18.0.0-7476.g705aae7f.el9.x86_64 5/33 2023-11-27T09:23:37.245 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7476.g705aae7f.el9. 6/33 2023-11-27T09:23:37.245 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-11-27T09:23:37.245 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-11-27T09:23:37.245 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2023-11-27T09:23:37.246 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2023-11-27T09:23:37.246 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2023-11-27T09:23:37.246 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2023-11-27T09:23:37.246 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2023-11-27T09:23:37.246 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2023-11-27T09:23:37.246 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 15/33 2023-11-27T09:23:37.246 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 16/33 2023-11-27T09:23:37.246 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-11-27T09:23:37.246 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-11-27T09:23:37.247 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-11-27T09:23:37.247 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-11-27T09:23:37.247 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-11-27T09:23:37.247 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-11-27T09:23:37.247 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-11-27T09:23:37.247 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-11-27T09:23:37.247 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-11-27T09:23:37.247 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-11-27T09:23:37.247 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-11-27T09:23:37.247 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-11-27T09:23:37.248 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-11-27T09:23:37.248 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-11-27T09:23:37.248 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-11-27T09:23:37.248 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-11-27T09:23:37.960 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-11-27T09:23:37.961 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:23:37.961 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-27T09:23:37.961 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:23:37.961 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:23:37.961 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:23:37.961 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core-2:18.0.0-7476.g705aae7f.el9.noarch 2023-11-27T09:23:37.961 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:23:37.961 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:23:37.962 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-11-27T09:23:37.962 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-11-27T09:23:37.962 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-11-27T09:23:37.962 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-11-27T09:23:37.962 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-11-27T09:23:37.962 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-11-27T09:23:37.962 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-11-27T09:23:37.962 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-11-27T09:23:37.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-11-27T09:23:37.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-11-27T09:23:37.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-11-27T09:23:37.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako-1.1.4-6.el9.noarch 2023-11-27T09:23:37.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-11-27T09:23:37.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-11-27T09:23:37.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-11-27T09:23:37.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ply-3.11-14.el9.noarch 2023-11-27T09:23:37.963 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend-3.1.0-2.el9.noarch 2023-11-27T09:23:37.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-11-27T09:23:37.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser-2.20-6.el9.noarch 2023-11-27T09:23:37.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz-2021.1-5.el9.noarch 2023-11-27T09:23:37.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-2.25.1-7.el9.noarch 2023-11-27T09:23:37.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-11-27T09:23:37.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-toml-0.10.2-6.el9.noarch 2023-11-27T09:23:37.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-11-27T09:23:37.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob-1.8.7-6.el9.noarch 2023-11-27T09:23:37.964 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-11-27T09:23:37.965 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-11-27T09:23:37.965 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:23:37.965 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:23:38.215 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-base 2023-11-27T09:23:38.832 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:05:23 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:23:38.932 INFO:teuthology.orchestra.run.smithi007.stdout:Package ceph-base-2:18.0.0-7476.g705aae7f.el9.x86_64 is already installed. 2023-11-27T09:23:38.985 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:23:38.986 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-11-27T09:23:38.986 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:23:39.072 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install cephadm 2023-11-27T09:23:39.669 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:05:24 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:23:39.818 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:23:39.819 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:23:39.819 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-11-27T09:23:39.819 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:23:39.819 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-27T09:23:39.819 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm noarch 2:18.0.0-7476.g705aae7f.el9 ceph-noarch 773 k 2023-11-27T09:23:39.819 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:23:39.819 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-27T09:23:39.819 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:23:39.820 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-27T09:23:39.820 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:23:39.820 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 773 k 2023-11-27T09:23:39.820 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 789 k 2023-11-27T09:23:39.820 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-27T09:23:40.893 INFO:teuthology.orchestra.run.smithi007.stdout:cephadm-18.0.0-7476.g705aae7f.el9.noarch.rpm 721 kB/s | 773 kB 00:01 2023-11-27T09:23:40.893 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-27T09:23:40.894 INFO:teuthology.orchestra.run.smithi007.stdout:Total 719 kB/s | 773 kB 00:01 2023-11-27T09:23:40.894 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-27T09:23:40.901 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-27T09:23:40.901 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-27T09:23:40.907 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-27T09:23:40.908 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-27T09:23:40.983 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-27T09:23:41.380 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephadm-2:18.0.0-7476.g705aae7f.el9.noarch 1/1 2023-11-27T09:23:41.476 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cephadm-2:18.0.0-7476.g705aae7f.el9.noarch 1/1 2023-11-27T09:23:41.854 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephadm-2:18.0.0-7476.g705aae7f.el9.noarch 1/1 2023-11-27T09:23:42.288 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cephadm-2:18.0.0-7476.g705aae7f.el9.noarch 1/1 2023-11-27T09:23:42.288 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:23:42.288 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-27T09:23:42.288 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm-2:18.0.0-7476.g705aae7f.el9.noarch 2023-11-27T09:23:42.288 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:23:42.288 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:23:42.539 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-immutable-object-cache 2023-11-27T09:23:43.154 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:05:28 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:23:43.304 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:23:43.304 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:23:43.304 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-27T09:23:43.304 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:23:43.304 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-27T09:23:43.305 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 143 k 2023-11-27T09:23:43.305 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:23:43.305 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-27T09:23:43.305 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:23:43.305 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-27T09:23:43.305 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:23:43.305 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 143 k 2023-11-27T09:23:43.305 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 437 k 2023-11-27T09:23:43.306 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-27T09:23:43.998 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-immutable-object-cache-18.0.0-7476.g705aae 207 kB/s | 143 kB 00:00 2023-11-27T09:23:43.999 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-27T09:23:43.999 INFO:teuthology.orchestra.run.smithi007.stdout:Total 206 kB/s | 143 kB 00:00 2023-11-27T09:23:43.999 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-27T09:23:44.008 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-27T09:23:44.008 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-27T09:23:44.066 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-27T09:23:44.066 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-27T09:23:44.209 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-27T09:23:44.274 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7476.g705aae7f. 1/1 2023-11-27T09:23:44.300 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7476.g705aae7f. 1/1 2023-11-27T09:23:44.300 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-27T09:23:44.300 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-11-27T09:23:44.300 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:23:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7476.g705aae7f. 1/1 2023-11-27T09:23:45.371 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:23:45.372 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-27T09:23:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:23:45.372 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:23:45.372 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:23:45.596 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr 2023-11-27T09:23:46.196 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:05:31 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:23:46.294 INFO:teuthology.orchestra.run.smithi007.stdout:Package ceph-mgr-2:18.0.0-7476.g705aae7f.el9.x86_64 is already installed. 2023-11-27T09:23:46.346 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:23:46.347 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-11-27T09:23:46.348 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:23:46.428 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-dashboard 2023-11-27T09:23:47.033 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:05:32 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:23:47.185 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:23:47.185 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:23:47.186 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-11-27T09:23:47.186 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:23:47.186 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-27T09:23:47.186 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7476.g705aae7f.el9 ceph-noarch 1.8 M 2023-11-27T09:23:47.186 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-11-27T09:23:47.187 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7476.g705aae7f.el9 ceph-noarch 28 k 2023-11-27T09:23:47.187 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7476.g705aae7f.el9 ceph-noarch 15 k 2023-11-27T09:23:47.187 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-11-27T09:23:47.187 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-11-27T09:23:47.187 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-11-27T09:23:47.187 INFO:teuthology.orchestra.run.smithi007.stdout:Installing weak dependencies: 2023-11-27T09:23:47.187 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-11-27T09:23:47.187 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:23:47.187 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-27T09:23:47.188 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:23:47.188 INFO:teuthology.orchestra.run.smithi007.stdout:Install 7 Packages 2023-11-27T09:23:47.188 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:23:47.189 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 2.1 M 2023-11-27T09:23:47.189 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 8.7 M 2023-11-27T09:23:47.189 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-27T09:23:47.827 INFO:teuthology.orchestra.run.smithi007.stdout:(1/7): ceph-prometheus-alerts-18.0.0-7476.g705a 32 kB/s | 15 kB 00:00 2023-11-27T09:23:47.911 INFO:teuthology.orchestra.run.smithi007.stdout:(2/7): ceph-grafana-dashboards-18.0.0-7476.g705 50 kB/s | 28 kB 00:00 2023-11-27T09:23:47.961 INFO:teuthology.orchestra.run.smithi007.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 68 kB/s | 9.0 kB 00:00 2023-11-27T09:23:48.012 INFO:teuthology.orchestra.run.smithi007.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 414 kB/s | 41 kB 00:00 2023-11-27T09:23:48.038 INFO:teuthology.orchestra.run.smithi007.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 408 kB/s | 31 kB 00:00 2023-11-27T09:23:48.095 INFO:teuthology.orchestra.run.smithi007.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2023-11-27T09:23:50.470 INFO:teuthology.orchestra.run.smithi007.stdout:(7/7): ceph-mgr-dashboard-18.0.0-7476.g705aae7f 578 kB/s | 1.8 MB 00:03 2023-11-27T09:23:50.479 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-27T09:23:50.479 INFO:teuthology.orchestra.run.smithi007.stdout:Total 642 kB/s | 2.1 MB 00:03 2023-11-27T09:23:50.479 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-27T09:23:50.488 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-27T09:23:50.489 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-27T09:23:50.565 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-27T09:23:50.565 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-27T09:23:50.875 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-27T09:23:50.930 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-11-27T09:23:50.964 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-11-27T09:23:51.011 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-11-27T09:23:51.059 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-11-27T09:23:51.089 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7476.g705aae7f.el9.n 5/7 2023-11-27T09:23:51.338 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7476.g705aae7f.el9. 6/7 2023-11-27T09:23:51.366 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7476.g705aae7f.el9.noarc 7/7 2023-11-27T09:23:51.915 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7476.g705aae7f.el9.noarc 7/7 2023-11-27T09:23:51.915 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7476.g705aae7f.el9. 1/7 2023-11-27T09:23:51.915 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7476.g705aae7f.el9.noarc 2/7 2023-11-27T09:23:51.915 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7476.g705aae7f.el9.n 3/7 2023-11-27T09:23:51.915 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-11-27T09:23:51.915 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-11-27T09:23:51.915 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-11-27T09:23:52.234 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-11-27T09:23:52.235 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:23:52.235 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-27T09:23:52.235 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards-2:18.0.0-7476.g705aae7f.el9.noarch 2023-11-27T09:23:52.235 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard-2:18.0.0-7476.g705aae7f.el9.noarch 2023-11-27T09:23:52.235 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts-2:18.0.0-7476.g705aae7f.el9.noarch 2023-11-27T09:23:52.235 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-11-27T09:23:52.235 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-11-27T09:23:52.235 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-11-27T09:23:52.236 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes-2.5.1-5.el9.noarch 2023-11-27T09:23:52.236 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:23:52.236 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:23:52.456 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-27T09:23:53.060 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:05:38 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:23:53.214 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:23:53.215 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2023-11-27T09:23:53.215 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-11-27T09:23:53.215 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2023-11-27T09:23:53.216 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-27T09:23:53.216 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7476.g705aae7f.el9 ceph-noarch 7.4 M 2023-11-27T09:23:53.216 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-11-27T09:23:53.216 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-11-27T09:23:53.216 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-11-27T09:23:53.216 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-11-27T09:23:53.216 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-11-27T09:23:53.216 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-11-27T09:23:53.216 INFO:teuthology.orchestra.run.smithi007.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-11-27T09:23:53.217 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-11-27T09:23:53.217 INFO:teuthology.orchestra.run.smithi007.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-11-27T09:23:53.217 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-11-27T09:23:53.217 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-11-27T09:23:53.217 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-11-27T09:23:53.217 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:23:53.217 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-27T09:23:53.217 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2023-11-27T09:23:53.218 INFO:teuthology.orchestra.run.smithi007.stdout:Install 12 Packages 2023-11-27T09:23:53.218 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:23:53.218 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 37 M 2023-11-27T09:23:53.218 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 209 M 2023-11-27T09:23:53.219 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-27T09:23:53.369 INFO:teuthology.orchestra.run.smithi007.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 296 kB/s | 32 kB 00:00 2023-11-27T09:23:53.419 INFO:teuthology.orchestra.run.smithi007.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 680 kB/s | 17 kB 00:00 2023-11-27T09:23:53.444 INFO:teuthology.orchestra.run.smithi007.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.4 MB/s | 34 kB 00:00 2023-11-27T09:23:53.512 INFO:teuthology.orchestra.run.smithi007.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2023-11-27T09:23:53.620 INFO:teuthology.orchestra.run.smithi007.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2023-11-27T09:23:53.695 INFO:teuthology.orchestra.run.smithi007.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 26 MB/s | 4.8 MB 00:00 2023-11-27T09:23:53.721 INFO:teuthology.orchestra.run.smithi007.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2023-11-27T09:23:53.796 INFO:teuthology.orchestra.run.smithi007.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 11 MB/s | 803 kB 00:00 2023-11-27T09:23:53.963 INFO:teuthology.orchestra.run.smithi007.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 59 MB/s | 16 MB 00:00 2023-11-27T09:23:53.988 INFO:teuthology.orchestra.run.smithi007.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2023-11-27T09:23:54.238 INFO:teuthology.orchestra.run.smithi007.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 894 kB/s | 245 kB 00:00 2023-11-27T09:24:07.387 INFO:teuthology.orchestra.run.smithi007.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-7 535 kB/s | 7.4 MB 00:14 2023-11-27T09:24:07.390 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-27T09:24:07.390 INFO:teuthology.orchestra.run.smithi007.stdout:Total 2.6 MB/s | 37 MB 00:14 2023-11-27T09:24:07.405 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-27T09:24:07.433 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-27T09:24:07.433 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-27T09:24:07.879 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-27T09:24:07.880 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-27T09:24:08.730 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-27T09:24:08.827 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-11-27T09:24:08.858 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-11-27T09:24:08.956 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-11-27T09:24:08.995 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-11-27T09:24:09.172 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-27T09:24:09.202 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-27T09:24:09.285 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-11-27T09:24:09.983 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-11-27T09:24:10.103 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-11-27T09:24:11.426 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-11-27T09:24:12.530 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-11-27T09:24:12.737 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7476.g705aa 12/12 2023-11-27T09:24:15.020 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7476.g705aa 12/12 2023-11-27T09:24:15.020 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7476.g705aa 1/12 2023-11-27T09:24:15.020 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-11-27T09:24:15.020 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-11-27T09:24:15.021 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-11-27T09:24:15.021 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-27T09:24:15.021 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-27T09:24:15.021 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-11-27T09:24:15.021 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-11-27T09:24:15.021 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-11-27T09:24:15.021 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-11-27T09:24:15.021 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-11-27T09:24:15.460 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-11-27T09:24:15.460 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:15.460 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-27T09:24:15.460 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7476.g705aae7f.el9.noarch 2023-11-27T09:24:15.460 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-11-27T09:24:15.460 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-11-27T09:24:15.461 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-11-27T09:24:15.461 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-11-27T09:24:15.461 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-11-27T09:24:15.461 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-0.3.21-2.el9.x86_64 2023-11-27T09:24:15.461 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-11-27T09:24:15.461 INFO:teuthology.orchestra.run.smithi007.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-11-27T09:24:15.461 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-11-27T09:24:15.461 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-11-27T09:24:15.461 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-11-27T09:24:15.461 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:15.461 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:24:15.864 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-rook 2023-11-27T09:24:16.467 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:01 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:24:16.621 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:24:16.622 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2023-11-27T09:24:16.622 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-11-27T09:24:16.622 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2023-11-27T09:24:16.622 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-27T09:24:16.622 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook noarch 2:18.0.0-7476.g705aae7f.el9 ceph-noarch 50 k 2023-11-27T09:24:16.623 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-11-27T09:24:16.623 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-11-27T09:24:16.623 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-11-27T09:24:16.623 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-11-27T09:24:16.623 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-11-27T09:24:16.623 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-11-27T09:24:16.623 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-11-27T09:24:16.623 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-11-27T09:24:16.623 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-11-27T09:24:16.623 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-11-27T09:24:16.624 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-11-27T09:24:16.624 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-11-27T09:24:16.624 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-11-27T09:24:16.624 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:16.624 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-27T09:24:16.624 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2023-11-27T09:24:16.624 INFO:teuthology.orchestra.run.smithi007.stdout:Install 13 Packages 2023-11-27T09:24:16.624 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:16.625 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 2.0 M 2023-11-27T09:24:16.625 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 26 M 2023-11-27T09:24:16.625 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-27T09:24:16.806 INFO:teuthology.orchestra.run.smithi007.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 243 kB/s | 24 kB 00:00 2023-11-27T09:24:16.856 INFO:teuthology.orchestra.run.smithi007.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 120 kB/s | 18 kB 00:00 2023-11-27T09:24:16.898 INFO:teuthology.orchestra.run.smithi007.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2023-11-27T09:24:16.915 INFO:teuthology.orchestra.run.smithi007.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2023-11-27T09:24:16.940 INFO:teuthology.orchestra.run.smithi007.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 5.1 MB/s | 211 kB 00:00 2023-11-27T09:24:16.966 INFO:teuthology.orchestra.run.smithi007.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2023-11-27T09:24:17.199 INFO:teuthology.orchestra.run.smithi007.stdout:(7/13): ceph-mgr-rook-18.0.0-7476.g705aae7f.el9 102 kB/s | 50 kB 00:00 2023-11-27T09:24:17.358 INFO:teuthology.orchestra.run.smithi007.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 38 kB/s | 15 kB 00:00 2023-11-27T09:24:17.400 INFO:teuthology.orchestra.run.smithi007.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 70 kB/s | 32 kB 00:00 2023-11-27T09:24:17.617 INFO:teuthology.orchestra.run.smithi007.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 273 kB/s | 59 kB 00:00 2023-11-27T09:24:17.800 INFO:teuthology.orchestra.run.smithi007.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 489 kB/s | 90 kB 00:00 2023-11-27T09:24:17.834 INFO:teuthology.orchestra.run.smithi007.stdout:(12/13): python3-google-auth-2.23.4-1.el9.noarc 312 kB/s | 197 kB 00:00 2023-11-27T09:24:17.951 INFO:teuthology.orchestra.run.smithi007.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2023-11-27T09:24:17.956 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-27T09:24:17.956 INFO:teuthology.orchestra.run.smithi007.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2023-11-27T09:24:17.957 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-27T09:24:17.973 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-27T09:24:17.973 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-27T09:24:18.087 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-27T09:24:18.088 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-27T09:24:18.617 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-27T09:24:18.807 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-11-27T09:24:18.870 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-11-27T09:24:18.927 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-11-27T09:24:18.962 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-11-27T09:24:18.996 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-11-27T09:24:19.093 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-11-27T09:24:19.202 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-11-27T09:24:19.265 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-11-27T09:24:20.030 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-11-27T09:24:20.086 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-11-27T09:24:20.120 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-11-27T09:24:20.168 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-11-27T09:24:20.198 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-rook-2:18.0.0-7476.g705aae7f.el9.noarch 13/13 2023-11-27T09:24:21.020 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7476.g705aae7f.el9.noarch 13/13 2023-11-27T09:24:21.020 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7476.g705aae7f.el9.noarch 1/13 2023-11-27T09:24:21.020 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-11-27T09:24:21.020 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-11-27T09:24:21.020 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-11-27T09:24:21.020 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-11-27T09:24:21.021 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-11-27T09:24:21.021 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-11-27T09:24:21.021 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-11-27T09:24:21.021 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-11-27T09:24:21.021 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-11-27T09:24:21.021 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-11-27T09:24:21.021 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-11-27T09:24:21.391 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-11-27T09:24:21.391 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:21.391 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-27T09:24:21.391 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook-2:18.0.0-7476.g705aae7f.el9.noarch 2023-11-27T09:24:21.391 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-11-27T09:24:21.391 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-11-27T09:24:21.392 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-11-27T09:24:21.392 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-11-27T09:24:21.392 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-11-27T09:24:21.392 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-11-27T09:24:21.392 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-11-27T09:24:21.392 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-11-27T09:24:21.392 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-11-27T09:24:21.392 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-11-27T09:24:21.392 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa-4.9-2.el9.noarch 2023-11-27T09:24:21.393 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-11-27T09:24:21.393 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:21.393 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:24:21.664 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-cephadm 2023-11-27T09:24:22.276 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:07 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:24:22.431 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:24:22.431 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2023-11-27T09:24:22.432 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-11-27T09:24:22.432 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2023-11-27T09:24:22.432 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-27T09:24:22.432 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7476.g705aae7f.el9 ceph-noarch 133 k 2023-11-27T09:24:22.432 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-11-27T09:24:22.432 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-11-27T09:24:22.432 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-11-27T09:24:22.433 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-11-27T09:24:22.433 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-11-27T09:24:22.433 INFO:teuthology.orchestra.run.smithi007.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-11-27T09:24:22.433 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:22.433 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-27T09:24:22.433 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2023-11-27T09:24:22.433 INFO:teuthology.orchestra.run.smithi007.stdout:Install 6 Packages 2023-11-27T09:24:22.433 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:22.434 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 6.8 M 2023-11-27T09:24:22.434 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 33 M 2023-11-27T09:24:22.434 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-27T09:24:22.954 INFO:teuthology.orchestra.run.smithi007.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2023-11-27T09:24:23.155 INFO:teuthology.orchestra.run.smithi007.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2023-11-27T09:24:23.446 INFO:teuthology.orchestra.run.smithi007.stdout:(3/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2023-11-27T09:24:23.472 INFO:teuthology.orchestra.run.smithi007.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 182 kB/s | 58 kB 00:00 2023-11-27T09:24:23.497 INFO:teuthology.orchestra.run.smithi007.stdout:(5/6): ceph-mgr-cephadm-18.0.0-7476.g705aae7f.e 190 kB/s | 133 kB 00:00 2023-11-27T09:24:23.522 INFO:teuthology.orchestra.run.smithi007.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 674 kB/s | 51 kB 00:00 2023-11-27T09:24:23.526 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-27T09:24:23.526 INFO:teuthology.orchestra.run.smithi007.stdout:Total 6.2 MB/s | 6.8 MB 00:01 2023-11-27T09:24:23.526 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-27T09:24:23.536 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-27T09:24:23.536 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-27T09:24:23.632 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-27T09:24:23.639 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-27T09:24:23.943 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-27T09:24:24.111 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-11-27T09:24:24.166 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-11-27T09:24:24.724 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-11-27T09:24:24.812 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-11-27T09:24:24.873 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-11-27T09:24:24.903 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7476.g705aae7f.el9.noarch 6/6 2023-11-27T09:24:25.770 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7476.g705aae7f.el9.noarch 6/6 2023-11-27T09:24:25.770 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7476.g705aae7f.el9.noarch 1/6 2023-11-27T09:24:25.770 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-11-27T09:24:25.770 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-11-27T09:24:25.770 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-11-27T09:24:25.770 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-11-27T09:24:26.079 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-11-27T09:24:26.079 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:26.079 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-27T09:24:26.079 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm-2:18.0.0-7476.g705aae7f.el9.noarch 2023-11-27T09:24:26.079 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-11-27T09:24:26.080 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel-2.9.1-2.el9.noarch 2023-11-27T09:24:26.080 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-11-27T09:24:26.080 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-11-27T09:24:26.080 INFO:teuthology.orchestra.run.smithi007.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-11-27T09:24:26.080 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:26.080 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:24:26.330 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-fuse 2023-11-27T09:24:26.936 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:11 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:24:27.089 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:24:27.090 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:24:27.090 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-11-27T09:24:27.090 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:24:27.090 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-27T09:24:27.090 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 833 k 2023-11-27T09:24:27.090 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-11-27T09:24:27.091 INFO:teuthology.orchestra.run.smithi007.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-11-27T09:24:27.091 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:27.091 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-27T09:24:27.091 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:24:27.091 INFO:teuthology.orchestra.run.smithi007.stdout:Install 2 Packages 2023-11-27T09:24:27.091 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:27.091 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 912 k 2023-11-27T09:24:27.091 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 3.0 M 2023-11-27T09:24:27.092 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-27T09:24:27.183 INFO:teuthology.orchestra.run.smithi007.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 852 kB/s | 79 kB 00:00 2023-11-27T09:24:29.533 INFO:teuthology.orchestra.run.smithi007.stdout:(2/2): ceph-fuse-18.0.0-7476.g705aae7f.el9.x86_ 341 kB/s | 833 kB 00:02 2023-11-27T09:24:29.534 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-27T09:24:29.535 INFO:teuthology.orchestra.run.smithi007.stdout:Total 373 kB/s | 912 kB 00:02 2023-11-27T09:24:29.535 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-27T09:24:29.561 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-27T09:24:29.561 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-27T09:24:29.645 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-27T09:24:29.645 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-27T09:24:29.836 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-27T09:24:29.979 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-11-27T09:24:30.052 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-fuse-2:18.0.0-7476.g705aae7f.el9.x86_64 2/2 2023-11-27T09:24:30.774 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7476.g705aae7f.el9.x86_64 2/2 2023-11-27T09:24:30.775 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-fuse-2:18.0.0-7476.g705aae7f.el9.x86_64 1/2 2023-11-27T09:24:31.147 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-11-27T09:24:31.148 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:31.148 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-27T09:24:31.148 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse-2:18.0.0-7476.g705aae7f.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-11-27T09:24:31.148 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:31.148 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:24:31.431 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-volume 2023-11-27T09:24:32.040 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:17 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:24:32.190 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:24:32.191 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:24:32.191 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-11-27T09:24:32.191 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:24:32.191 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-27T09:24:32.191 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume noarch 2:18.0.0-7476.g705aae7f.el9 ceph-noarch 262 k 2023-11-27T09:24:32.191 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:32.191 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-27T09:24:32.191 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:24:32.192 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-27T09:24:32.192 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:32.192 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 262 k 2023-11-27T09:24:32.192 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 1.2 M 2023-11-27T09:24:32.192 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-27T09:24:32.930 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-volume-18.0.0-7476.g705aae7f.el9.noarch.rp 355 kB/s | 262 kB 00:00 2023-11-27T09:24:32.930 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-27T09:24:32.931 INFO:teuthology.orchestra.run.smithi007.stdout:Total 355 kB/s | 262 kB 00:00 2023-11-27T09:24:32.931 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-27T09:24:32.940 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-27T09:24:32.940 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-27T09:24:32.981 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-27T09:24:32.981 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-27T09:24:33.209 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-27T09:24:33.281 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-volume-2:18.0.0-7476.g705aae7f.el9.noarch 1/1 2023-11-27T09:24:33.298 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-volume-2:18.0.0-7476.g705aae7f.el9.noarch 1/1 2023-11-27T09:24:33.298 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-27T09:24:33.298 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-11-27T09:24:33.298 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:34.402 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-volume-2:18.0.0-7476.g705aae7f.el9.noarch 1/1 2023-11-27T09:24:34.402 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:34.402 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-27T09:24:34.402 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume-2:18.0.0-7476.g705aae7f.el9.noarch 2023-11-27T09:24:34.402 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:34.402 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:24:34.731 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados-devel 2023-11-27T09:24:35.345 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:20 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:24:35.497 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:24:35.498 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:24:35.498 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-27T09:24:35.498 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:24:35.498 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-27T09:24:35.498 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 127 k 2023-11-27T09:24:35.499 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:35.499 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-27T09:24:35.499 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:24:35.499 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-27T09:24:35.499 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:35.499 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 127 k 2023-11-27T09:24:35.499 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 456 k 2023-11-27T09:24:35.499 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-27T09:24:36.210 INFO:teuthology.orchestra.run.smithi007.stdout:librados-devel-18.0.0-7476.g705aae7f.el9.x86_64 178 kB/s | 127 kB 00:00 2023-11-27T09:24:36.211 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-27T09:24:36.211 INFO:teuthology.orchestra.run.smithi007.stdout:Total 178 kB/s | 127 kB 00:00 2023-11-27T09:24:36.211 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-27T09:24:36.217 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-27T09:24:36.217 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-27T09:24:36.277 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-27T09:24:36.277 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-27T09:24:36.432 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-27T09:24:36.537 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librados-devel-2:18.0.0-7476.g705aae7f.el9.x86_64 1/1 2023-11-27T09:24:36.978 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados-devel-2:18.0.0-7476.g705aae7f.el9.x86_64 1/1 2023-11-27T09:24:37.294 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados-devel-2:18.0.0-7476.g705aae7f.el9.x86_64 1/1 2023-11-27T09:24:37.294 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:37.294 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-27T09:24:37.294 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:24:37.295 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:37.295 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:24:37.550 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install libcephfs2 2023-11-27T09:24:38.157 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:23 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:24:38.255 INFO:teuthology.orchestra.run.smithi007.stdout:Package libcephfs2-2:18.0.0-7476.g705aae7f.el9.x86_64 is already installed. 2023-11-27T09:24:38.307 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:24:38.309 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-11-27T09:24:38.309 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:24:38.385 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install libcephfs-devel 2023-11-27T09:24:38.993 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:23 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:24:39.144 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:24:39.144 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:24:39.145 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-27T09:24:39.145 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:24:39.145 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-27T09:24:39.145 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 31 k 2023-11-27T09:24:39.145 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:39.145 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-27T09:24:39.145 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:24:39.145 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-27T09:24:39.146 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:39.146 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 31 k 2023-11-27T09:24:39.146 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 139 k 2023-11-27T09:24:39.146 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-27T09:24:39.590 INFO:teuthology.orchestra.run.smithi007.stdout:libcephfs-devel-18.0.0-7476.g705aae7f.el9.x86_6 70 kB/s | 31 kB 00:00 2023-11-27T09:24:39.591 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-27T09:24:39.591 INFO:teuthology.orchestra.run.smithi007.stdout:Total 70 kB/s | 31 kB 00:00 2023-11-27T09:24:39.591 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-27T09:24:39.594 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-27T09:24:39.595 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-27T09:24:39.620 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-27T09:24:39.621 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-27T09:24:39.719 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-27T09:24:39.840 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs-devel-2:18.0.0-7476.g705aae7f.el9.x86_64 1/1 2023-11-27T09:24:40.295 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7476.g705aae7f.el9.x86_64 1/1 2023-11-27T09:24:40.577 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs-devel-2:18.0.0-7476.g705aae7f.el9.x86_64 1/1 2023-11-27T09:24:40.577 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:40.577 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-27T09:24:40.577 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:24:40.577 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:40.577 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:24:40.758 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados2 2023-11-27T09:24:41.358 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:26 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:24:41.454 INFO:teuthology.orchestra.run.smithi007.stdout:Package librados2-2:18.0.0-7476.g705aae7f.el9.x86_64 is already installed. 2023-11-27T09:24:41.507 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:24:41.508 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-11-27T09:24:41.508 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:24:41.582 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librbd1 2023-11-27T09:24:42.189 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:27 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:24:42.285 INFO:teuthology.orchestra.run.smithi007.stdout:Package librbd1-2:18.0.0-7476.g705aae7f.el9.x86_64 is already installed. 2023-11-27T09:24:42.339 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:24:42.340 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-11-27T09:24:42.340 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:24:42.413 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rados 2023-11-27T09:24:43.016 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:28 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:24:43.114 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rados-2:18.0.0-7476.g705aae7f.el9.x86_64 is already installed. 2023-11-27T09:24:43.166 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:24:43.168 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-11-27T09:24:43.168 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:24:43.238 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rgw 2023-11-27T09:24:43.853 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:28 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:24:43.953 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rgw-2:18.0.0-7476.g705aae7f.el9.x86_64 is already installed. 2023-11-27T09:24:44.007 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:24:44.008 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-11-27T09:24:44.008 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:24:44.083 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-cephfs 2023-11-27T09:24:44.701 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:29 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:24:44.801 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-cephfs-2:18.0.0-7476.g705aae7f.el9.x86_64 is already installed. 2023-11-27T09:24:44.854 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:24:44.855 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-11-27T09:24:44.855 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:24:44.929 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rbd 2023-11-27T09:24:45.532 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:30 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:24:45.630 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rbd-2:18.0.0-7476.g705aae7f.el9.x86_64 is already installed. 2023-11-27T09:24:45.686 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:24:45.687 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2023-11-27T09:24:45.687 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:24:45.762 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-fuse 2023-11-27T09:24:46.361 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:31 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:24:46.510 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:24:46.510 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:24:46.510 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2023-11-27T09:24:46.511 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:24:46.511 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-27T09:24:46.511 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 86 k 2023-11-27T09:24:46.511 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:46.511 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-27T09:24:46.511 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:24:46.511 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-27T09:24:46.511 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:46.512 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 86 k 2023-11-27T09:24:46.512 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 256 k 2023-11-27T09:24:46.512 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-27T09:24:47.086 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-fuse-18.0.0-7476.g705aae7f.el9.x86_64.rpm 150 kB/s | 86 kB 00:00 2023-11-27T09:24:47.086 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-27T09:24:47.086 INFO:teuthology.orchestra.run.smithi007.stdout:Total 149 kB/s | 86 kB 00:00 2023-11-27T09:24:47.087 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-27T09:24:47.094 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-27T09:24:47.095 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-27T09:24:47.154 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-27T09:24:47.154 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-27T09:24:47.299 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-27T09:24:47.436 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-fuse-2:18.0.0-7476.g705aae7f.el9.x86_64 1/1 2023-11-27T09:24:47.873 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7476.g705aae7f.el9.x86_64 1/1 2023-11-27T09:24:48.201 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-fuse-2:18.0.0-7476.g705aae7f.el9.x86_64 1/1 2023-11-27T09:24:48.201 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:48.201 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-27T09:24:48.201 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:24:48.201 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:48.202 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:24:48.472 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-mirror 2023-11-27T09:24:49.077 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:34 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:24:49.227 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:24:49.228 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:24:49.228 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2023-11-27T09:24:49.228 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:24:49.228 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-27T09:24:49.228 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 3.0 M 2023-11-27T09:24:49.228 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:49.229 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-27T09:24:49.229 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:24:49.229 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-27T09:24:49.229 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:49.229 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 3.0 M 2023-11-27T09:24:49.229 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 13 M 2023-11-27T09:24:49.229 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-27T09:24:54.414 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-mirror-18.0.0-7476.g705aae7f.el9.x86_64.rpm 590 kB/s | 3.0 MB 00:05 2023-11-27T09:24:54.415 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-27T09:24:54.416 INFO:teuthology.orchestra.run.smithi007.stdout:Total 590 kB/s | 3.0 MB 00:05 2023-11-27T09:24:54.416 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-27T09:24:54.427 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-27T09:24:54.427 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-27T09:24:54.505 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-27T09:24:54.506 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-27T09:24:54.949 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-27T09:24:55.015 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-mirror-2:18.0.0-7476.g705aae7f.el9.x86_64 1/1 2023-11-27T09:24:55.041 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7476.g705aae7f.el9.x86_64 1/1 2023-11-27T09:24:55.041 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-27T09:24:55.041 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-11-27T09:24:55.041 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-27T09:24:55.041 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-27T09:24:55.041 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:56.273 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-mirror-2:18.0.0-7476.g705aae7f.el9.x86_64 1/1 2023-11-27T09:24:56.273 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:56.274 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-27T09:24:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:24:56.274 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:56.274 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:24:56.503 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-nbd 2023-11-27T09:24:57.108 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:42 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:24:57.258 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:24:57.258 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:24:57.258 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2023-11-27T09:24:57.258 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:24:57.258 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-27T09:24:57.259 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 169 k 2023-11-27T09:24:57.259 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:57.259 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-27T09:24:57.259 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:24:57.259 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2023-11-27T09:24:57.259 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:57.260 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 169 k 2023-11-27T09:24:57.260 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 536 k 2023-11-27T09:24:57.260 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-27T09:24:58.235 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-nbd-18.0.0-7476.g705aae7f.el9.x86_64.rpm 173 kB/s | 169 kB 00:00 2023-11-27T09:24:58.236 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-27T09:24:58.237 INFO:teuthology.orchestra.run.smithi007.stdout:Total 173 kB/s | 169 kB 00:00 2023-11-27T09:24:58.237 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-27T09:24:58.247 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-27T09:24:58.247 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-27T09:24:58.310 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-27T09:24:58.311 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-27T09:24:58.477 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-27T09:24:58.582 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-nbd-2:18.0.0-7476.g705aae7f.el9.x86_64 1/1 2023-11-27T09:24:59.030 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7476.g705aae7f.el9.x86_64 1/1 2023-11-27T09:24:59.315 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-nbd-2:18.0.0-7476.g705aae7f.el9.x86_64 1/1 2023-11-27T09:24:59.316 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:59.316 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-27T09:24:59.316 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:24:59.316 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:24:59.316 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:24:59.564 DEBUG:teuthology.parallel:result is None 2023-11-27T09:24:59.565 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3 2023-11-27T09:24:59.780 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-27T09:24:59.803 INFO:teuthology.orchestra.run.smithi007.stdout:18.0.0-7476.g705aae7f.el9 2023-11-27T09:24:59.803 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7476.g705aae7f.el9 2023-11-27T09:24:59.803 INFO:teuthology.task.install:The correct ceph version 18.0.0-7476.g705aae7f is installed. 2023-11-27T09:24:59.804 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-27T09:24:59.804 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:24:59.804 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-27T09:24:59.877 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-27T09:24:59.877 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:24:59.877 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/daemon-helper 2023-11-27T09:24:59.948 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-27T09:25:00.020 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-27T09:25:00.020 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:25:00.020 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-27T09:25:00.092 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-27T09:25:00.164 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-27T09:25:00.164 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:25:00.164 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/stdin-killer 2023-11-27T09:25:00.235 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-27T09:25:00.311 INFO:teuthology.run_tasks:Running task ceph... 2023-11-27T09:25:00.372 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-27T09:25:00.372 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 777 /var/log/ceph 2023-11-27T09:25:00.402 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-27T09:25:00.403 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-27T09:25:00.472 INFO:tasks.ceph:Creating extra log directories... 2023-11-27T09:25:00.473 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-27T09:25:00.550 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-11-27T09:25:00.550 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-27T09:25:00.551 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467536', 'branch': 'wip-yingxin-crimson-make-crosscore-send-ordered', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_rand_read}', 'email': 'yingxin.cheng@intel.com', 'first_in_suite': False, 'job_id': '7467536', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yingxin-crimson-make-crosscore-send-ordered'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3'}}, 'workunit': {'branch': 'wip-yingxin-crimson-make-crosscore-send-ordered', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3'}}, 'owner': 'scheduled_yingxin@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'sha1': '705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-yingxin-crimson-make-crosscore-send-ordered', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3', 'targets': {'smithi007.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIo1oSgQ96OcqSO1+jWyD2Ss+I4+QETc12xGTgB8SwCMRoGMHUTdGkPL3ekuNjOA5AelGYNkpwSR4suTNnzD0VY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '2442239d2653456406c25ae0c71b689c8f2657b6', 'timestamp': '2023-11-27_09:03:31', 'tube': 'smithi', 'user': 'yingxin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-11-27T09:25:00.551 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-27T09:25:00.599 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/run/ceph 2023-11-27T09:25:00.670 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:25:00.670 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2023-11-27T09:25:00.725 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-27T09:25:00.725 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 2023-11-27T09:25:00.780 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-27T09:25:00.781 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-27T09:25:00.781 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 664 Links: 1 2023-11-27T09:25:00.781 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-27T09:25:00.781 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2023-11-27T09:25:00.781 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-11-27 09:24:55.296542810 +0000 2023-11-27T09:25:00.781 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-11-27 09:17:20.728978884 +0000 2023-11-27T09:25:00.781 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-11-27 09:17:20.728978884 +0000 2023-11-27T09:25:00.781 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2023-11-27 09:17:20.728978884 +0000 2023-11-27T09:25:00.781 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-27T09:25:00.849 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-11-27T09:25:00.849 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-11-27T09:25:00.849 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 9.5147e-05 s, 5.4 MB/s 2023-11-27T09:25:00.850 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-27T09:25:00.906 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 2023-11-27T09:25:00.961 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-27T09:25:00.962 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-27T09:25:00.962 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 681 Links: 1 2023-11-27T09:25:00.962 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-27T09:25:00.962 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2023-11-27T09:25:00.962 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-11-27 09:24:55.296542810 +0000 2023-11-27T09:25:00.962 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-11-27 09:17:21.288970415 +0000 2023-11-27T09:25:00.962 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-11-27 09:17:21.288970415 +0000 2023-11-27T09:25:00.962 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2023-11-27 09:17:21.288970415 +0000 2023-11-27T09:25:00.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-27T09:25:01.030 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-11-27T09:25:01.030 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-11-27T09:25:01.030 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 9.2963e-05 s, 5.5 MB/s 2023-11-27T09:25:01.032 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-27T09:25:01.088 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 2023-11-27T09:25:01.143 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-27T09:25:01.143 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-27T09:25:01.143 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 702 Links: 1 2023-11-27T09:25:01.143 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-27T09:25:01.143 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2023-11-27T09:25:01.143 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-11-27 09:24:55.296542810 +0000 2023-11-27T09:25:01.143 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-11-27 09:17:21.851961899 +0000 2023-11-27T09:25:01.143 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-11-27 09:17:21.851961899 +0000 2023-11-27T09:25:01.144 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2023-11-27 09:17:21.851961899 +0000 2023-11-27T09:25:01.144 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-27T09:25:01.211 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-11-27T09:25:01.212 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-11-27T09:25:01.212 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 7.7337e-05 s, 6.6 MB/s 2023-11-27T09:25:01.213 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-27T09:25:01.269 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 2023-11-27T09:25:01.325 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-27T09:25:01.325 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-27T09:25:01.325 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 720 Links: 1 2023-11-27T09:25:01.325 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-27T09:25:01.325 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2023-11-27T09:25:01.325 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2023-11-27 09:24:55.297542794 +0000 2023-11-27T09:25:01.325 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2023-11-27 09:17:22.428953172 +0000 2023-11-27T09:25:01.325 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2023-11-27 09:17:22.428953172 +0000 2023-11-27T09:25:01.325 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2023-11-27 09:17:22.428953172 +0000 2023-11-27T09:25:01.326 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-27T09:25:01.393 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2023-11-27T09:25:01.393 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2023-11-27T09:25:01.393 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 9.4708e-05 s, 5.4 MB/s 2023-11-27T09:25:01.394 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-27T09:25:01.450 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-27T09:25:01.451 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-27T09:25:01.451 INFO:tasks.ceph:Generating config... 2023-11-27T09:25:01.452 INFO:tasks.ceph:[osd] debug monc = 20 2023-11-27T09:25:01.452 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-27T09:25:01.452 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2023-11-27T09:25:01.452 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-11-27T09:25:01.452 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-27T09:25:01.452 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2023-11-27T09:25:01.452 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2023-11-27T09:25:01.452 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-11-27T09:25:01.452 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-11-27T09:25:01.452 INFO:tasks.ceph:[osd] debug filestore = 0/0 2023-11-27T09:25:01.452 INFO:tasks.ceph:[osd] debug journal = 0/0 2023-11-27T09:25:01.453 INFO:tasks.ceph:[osd] debug ms = 0/0 2023-11-27T09:25:01.453 INFO:tasks.ceph:[osd] debug osd = 0/0 2023-11-27T09:25:01.453 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-11-27T09:25:01.453 INFO:tasks.ceph:[osd] err to stderr = True 2023-11-27T09:25:01.453 INFO:tasks.ceph:[osd] log flush on exit = True 2023-11-27T09:25:01.453 INFO:tasks.ceph:[osd] log to file = False 2023-11-27T09:25:01.453 INFO:tasks.ceph:[osd] log to stderr = True 2023-11-27T09:25:01.453 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-27T09:25:01.453 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-27T09:25:01.453 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-27T09:25:01.453 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-27T09:25:01.453 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-27T09:25:01.453 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-11-27T09:25:01.453 INFO:tasks.ceph:[global] auth client required = none 2023-11-27T09:25:01.454 INFO:tasks.ceph:[global] auth cluster required = none 2023-11-27T09:25:01.454 INFO:tasks.ceph:[global] auth service required = none 2023-11-27T09:25:01.454 INFO:tasks.ceph:[global] auth supported = none 2023-11-27T09:25:01.454 INFO:tasks.ceph:[global] debug asok = 0/0 2023-11-27T09:25:01.454 INFO:tasks.ceph:[global] debug auth = 0/0 2023-11-27T09:25:01.454 INFO:tasks.ceph:[global] debug bdev = 0/0 2023-11-27T09:25:01.454 INFO:tasks.ceph:[global] debug bluefs = 0/0 2023-11-27T09:25:01.454 INFO:tasks.ceph:[global] debug bluestore = 0/0 2023-11-27T09:25:01.454 INFO:tasks.ceph:[global] debug buffer = 0/0 2023-11-27T09:25:01.454 INFO:tasks.ceph:[global] debug civetweb = 0/0 2023-11-27T09:25:01.454 INFO:tasks.ceph:[global] debug client = 0/0 2023-11-27T09:25:01.454 INFO:tasks.ceph:[global] debug compressor = 0/0 2023-11-27T09:25:01.454 INFO:tasks.ceph:[global] debug context = 0/0 2023-11-27T09:25:01.455 INFO:tasks.ceph:[global] debug crush = 0/0 2023-11-27T09:25:01.455 INFO:tasks.ceph:[global] debug crypto = 0/0 2023-11-27T09:25:01.455 INFO:tasks.ceph:[global] debug dpdk = 0/0 2023-11-27T09:25:01.455 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2023-11-27T09:25:01.455 INFO:tasks.ceph:[global] debug filer = 0/0 2023-11-27T09:25:01.455 INFO:tasks.ceph:[global] debug filestore = 0/0 2023-11-27T09:25:01.455 INFO:tasks.ceph:[global] debug finisher = 0/0 2023-11-27T09:25:01.455 INFO:tasks.ceph:[global] debug fuse = 0/0 2023-11-27T09:25:01.455 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2023-11-27T09:25:01.455 INFO:tasks.ceph:[global] debug javaclient = 0/0 2023-11-27T09:25:01.455 INFO:tasks.ceph:[global] debug journal = 0/0 2023-11-27T09:25:01.455 INFO:tasks.ceph:[global] debug journaler = 0/0 2023-11-27T09:25:01.455 INFO:tasks.ceph:[global] debug kstore = 0/0 2023-11-27T09:25:01.456 INFO:tasks.ceph:[global] debug leveldb = 0/0 2023-11-27T09:25:01.456 INFO:tasks.ceph:[global] debug lockdep = 0/0 2023-11-27T09:25:01.456 INFO:tasks.ceph:[global] debug mds = 0/0 2023-11-27T09:25:01.456 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2023-11-27T09:25:01.456 INFO:tasks.ceph:[global] debug mds locker = 0/0 2023-11-27T09:25:01.456 INFO:tasks.ceph:[global] debug mds log = 0/0 2023-11-27T09:25:01.456 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2023-11-27T09:25:01.456 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2023-11-27T09:25:01.456 INFO:tasks.ceph:[global] debug memdb = 0/0 2023-11-27T09:25:01.456 INFO:tasks.ceph:[global] debug mgr = 0/0 2023-11-27T09:25:01.456 INFO:tasks.ceph:[global] debug mgrc = 0/0 2023-11-27T09:25:01.456 INFO:tasks.ceph:[global] debug mon = 0/0 2023-11-27T09:25:01.456 INFO:tasks.ceph:[global] debug monc = 0/0 2023-11-27T09:25:01.456 INFO:tasks.ceph:[global] debug ms = 0/0 2023-11-27T09:25:01.457 INFO:tasks.ceph:[global] debug objclass = 0/0 2023-11-27T09:25:01.457 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2023-11-27T09:25:01.457 INFO:tasks.ceph:[global] debug objecter = 0/0 2023-11-27T09:25:01.457 INFO:tasks.ceph:[global] debug optracker = 0/0 2023-11-27T09:25:01.457 INFO:tasks.ceph:[global] debug osd = 0/0 2023-11-27T09:25:01.457 INFO:tasks.ceph:[global] debug paxos = 0/0 2023-11-27T09:25:01.457 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2023-11-27T09:25:01.457 INFO:tasks.ceph:[global] debug rados = 0/0 2023-11-27T09:25:01.457 INFO:tasks.ceph:[global] debug rbd = 0/0 2023-11-27T09:25:01.457 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2023-11-27T09:25:01.457 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2023-11-27T09:25:01.457 INFO:tasks.ceph:[global] debug refs = 0/0 2023-11-27T09:25:01.457 INFO:tasks.ceph:[global] debug rgw = 0/0 2023-11-27T09:25:01.457 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2023-11-27T09:25:01.458 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2023-11-27T09:25:01.458 INFO:tasks.ceph:[global] debug striper = 0/0 2023-11-27T09:25:01.458 INFO:tasks.ceph:[global] debug throttle = 0/0 2023-11-27T09:25:01.458 INFO:tasks.ceph:[global] debug timer = 0/0 2023-11-27T09:25:01.458 INFO:tasks.ceph:[global] debug tp = 0/0 2023-11-27T09:25:01.458 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2023-11-27T09:25:01.458 INFO:tasks.ceph:[global] ms client mode = crc 2023-11-27T09:25:01.458 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-11-27T09:25:01.458 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-11-27T09:25:01.458 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-11-27T09:25:01.458 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-11-27T09:25:01.458 INFO:tasks.ceph:[global] ms service mode = crc 2023-11-27T09:25:01.458 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-27T09:25:01.459 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-27T09:25:01.459 INFO:tasks.ceph:[mon] debug mon = 0/0 2023-11-27T09:25:01.459 INFO:tasks.ceph:[mon] debug ms = 0/0 2023-11-27T09:25:01.459 INFO:tasks.ceph:[mon] debug paxos = 0/0 2023-11-27T09:25:01.459 INFO:tasks.ceph:[mon] osd pool default crimson = True 2023-11-27T09:25:01.459 INFO:tasks.ceph:Setting up mon.a... 2023-11-27T09:25:01.459 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-11-27T09:25:01.546 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/ceph.keyring 2023-11-27T09:25:01.549 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-11-27T09:25:01.602 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-27T09:25:01.631 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.7')] 2023-11-27T09:25:01.632 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.7', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-11-27T09:25:01.633 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:25:01.633 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-27T09:25:01.686 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.7 --print /home/ubuntu/cephtest/ceph.monmap 2023-11-27T09:25:01.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:01.771+0000 7f97c0859e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:01.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:01.771+0000 7f97c0859e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:01.772 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:01.771+0000 7f97c0859e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:01.773 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-11-27T09:25:01.773 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: generated fsid ffcd362b-f2f8-4720-b3d4-ddcd1c2d0a43 2023-11-27T09:25:01.773 INFO:teuthology.orchestra.run.smithi007.stdout:setting min_mon_release = pacific 2023-11-27T09:25:01.773 INFO:teuthology.orchestra.run.smithi007.stdout:epoch 0 2023-11-27T09:25:01.773 INFO:teuthology.orchestra.run.smithi007.stdout:fsid ffcd362b-f2f8-4720-b3d4-ddcd1c2d0a43 2023-11-27T09:25:01.773 INFO:teuthology.orchestra.run.smithi007.stdout:last_changed 2023-11-27T09:25:01.773019+0000 2023-11-27T09:25:01.774 INFO:teuthology.orchestra.run.smithi007.stdout:created 2023-11-27T09:25:01.773019+0000 2023-11-27T09:25:01.774 INFO:teuthology.orchestra.run.smithi007.stdout:min_mon_release 16 (pacific) 2023-11-27T09:25:01.774 INFO:teuthology.orchestra.run.smithi007.stdout:election_strategy: 1 2023-11-27T09:25:01.774 INFO:teuthology.orchestra.run.smithi007.stdout:0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2023-11-27T09:25:01.774 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-11-27T09:25:01.775 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-27T09:25:01.790 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ffcd362b-f2f8-4720-b3d4-ddcd1c2d0a43... 2023-11-27T09:25:01.791 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-27T09:25:01.890 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2023-11-27T09:25:01.891 INFO:teuthology.orchestra.run.smithi007.stdout: chdir = "" 2023-11-27T09:25:01.891 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-27T09:25:01.891 INFO:teuthology.orchestra.run.smithi007.stdout: auth supported = none 2023-11-27T09:25:01.891 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.891 INFO:teuthology.orchestra.run.smithi007.stdout: filestore xattr use omap = true 2023-11-27T09:25:01.891 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.891 INFO:teuthology.orchestra.run.smithi007.stdout: mon clock drift allowed = 1.000 2023-11-27T09:25:01.891 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.891 INFO:teuthology.orchestra.run.smithi007.stdout: osd crush chooseleaf type = 0 2023-11-27T09:25:01.891 INFO:teuthology.orchestra.run.smithi007.stdout: auth debug = true 2023-11-27T09:25:01.891 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.891 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on old message = true 2023-11-27T09:25:01.891 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on bug = true 2023-11-27T09:25:01.891 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.892 INFO:teuthology.orchestra.run.smithi007.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-27T09:25:01.892 INFO:teuthology.orchestra.run.smithi007.stdout: mon pg warn max object skew = 0 2023-11-27T09:25:01.892 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.892 INFO:teuthology.orchestra.run.smithi007.stdout: # disable pg_autoscaler by default for new pools 2023-11-27T09:25:01.892 INFO:teuthology.orchestra.run.smithi007.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-27T09:25:01.892 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.892 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default size = 2 2023-11-27T09:25:01.892 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.892 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow primary affinity = true 2023-11-27T09:25:01.892 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow pg remap = true 2023-11-27T09:25:01.892 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on legacy crush tunables = false 2023-11-27T09:25:01.892 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on crush straw calc version zero = false 2023-11-27T09:25:01.892 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on no sortbitwise = false 2023-11-27T09:25:01.892 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on osd down out interval zero = false 2023-11-27T09:25:01.893 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on too few osds = false 2023-11-27T09:25:01.893 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-27T09:25:01.893 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-27T09:25:01.893 INFO:teuthology.orchestra.run.smithi007.stdout: mon_allow_pool_size_one = true 2023-11-27T09:25:01.893 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.893 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-27T09:25:01.893 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.893 INFO:teuthology.orchestra.run.smithi007.stdout: osd default data pool replay window = 5 2023-11-27T09:25:01.893 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.893 INFO:teuthology.orchestra.run.smithi007.stdout: mon allow pool delete = true 2023-11-27T09:25:01.893 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.893 INFO:teuthology.orchestra.run.smithi007.stdout: mon cluster log file level = debug 2023-11-27T09:25:01.893 INFO:teuthology.orchestra.run.smithi007.stdout: debug asserts on shutdown = true 2023-11-27T09:25:01.893 INFO:teuthology.orchestra.run.smithi007.stdout: mon health detail to clog = false 2023-11-27T09:25:01.894 INFO:teuthology.orchestra.run.smithi007.stdout: mon host = 172.21.15.7 2023-11-27T09:25:01.894 INFO:teuthology.orchestra.run.smithi007.stdout: auth client required = none 2023-11-27T09:25:01.894 INFO:teuthology.orchestra.run.smithi007.stdout: auth cluster required = none 2023-11-27T09:25:01.894 INFO:teuthology.orchestra.run.smithi007.stdout: auth service required = none 2023-11-27T09:25:01.894 INFO:teuthology.orchestra.run.smithi007.stdout: debug asok = 0/0 2023-11-27T09:25:01.894 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 0/0 2023-11-27T09:25:01.894 INFO:teuthology.orchestra.run.smithi007.stdout: debug bdev = 0/0 2023-11-27T09:25:01.894 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluefs = 0/0 2023-11-27T09:25:01.894 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluestore = 0/0 2023-11-27T09:25:01.894 INFO:teuthology.orchestra.run.smithi007.stdout: debug buffer = 0/0 2023-11-27T09:25:01.895 INFO:teuthology.orchestra.run.smithi007.stdout: debug civetweb = 0/0 2023-11-27T09:25:01.895 INFO:teuthology.orchestra.run.smithi007.stdout: debug client = 0/0 2023-11-27T09:25:01.895 INFO:teuthology.orchestra.run.smithi007.stdout: debug compressor = 0/0 2023-11-27T09:25:01.895 INFO:teuthology.orchestra.run.smithi007.stdout: debug context = 0/0 2023-11-27T09:25:01.895 INFO:teuthology.orchestra.run.smithi007.stdout: debug crush = 0/0 2023-11-27T09:25:01.895 INFO:teuthology.orchestra.run.smithi007.stdout: debug crypto = 0/0 2023-11-27T09:25:01.895 INFO:teuthology.orchestra.run.smithi007.stdout: debug dpdk = 0/0 2023-11-27T09:25:01.895 INFO:teuthology.orchestra.run.smithi007.stdout: debug eventtrace = 0/0 2023-11-27T09:25:01.895 INFO:teuthology.orchestra.run.smithi007.stdout: debug filer = 0/0 2023-11-27T09:25:01.895 INFO:teuthology.orchestra.run.smithi007.stdout: debug filestore = 0/0 2023-11-27T09:25:01.895 INFO:teuthology.orchestra.run.smithi007.stdout: debug finisher = 0/0 2023-11-27T09:25:01.895 INFO:teuthology.orchestra.run.smithi007.stdout: debug fuse = 0/0 2023-11-27T09:25:01.895 INFO:teuthology.orchestra.run.smithi007.stdout: debug heartbeatmap = 0/0 2023-11-27T09:25:01.896 INFO:teuthology.orchestra.run.smithi007.stdout: debug javaclient = 0/0 2023-11-27T09:25:01.896 INFO:teuthology.orchestra.run.smithi007.stdout: debug journal = 0/0 2023-11-27T09:25:01.896 INFO:teuthology.orchestra.run.smithi007.stdout: debug journaler = 0/0 2023-11-27T09:25:01.896 INFO:teuthology.orchestra.run.smithi007.stdout: debug kstore = 0/0 2023-11-27T09:25:01.896 INFO:teuthology.orchestra.run.smithi007.stdout: debug leveldb = 0/0 2023-11-27T09:25:01.896 INFO:teuthology.orchestra.run.smithi007.stdout: debug lockdep = 0/0 2023-11-27T09:25:01.896 INFO:teuthology.orchestra.run.smithi007.stdout: debug mds = 0/0 2023-11-27T09:25:01.896 INFO:teuthology.orchestra.run.smithi007.stdout: debug mds balancer = 0/0 2023-11-27T09:25:01.896 INFO:teuthology.orchestra.run.smithi007.stdout: debug mds locker = 0/0 2023-11-27T09:25:01.896 INFO:teuthology.orchestra.run.smithi007.stdout: debug mds log = 0/0 2023-11-27T09:25:01.896 INFO:teuthology.orchestra.run.smithi007.stdout: debug mds log expire = 0/0 2023-11-27T09:25:01.896 INFO:teuthology.orchestra.run.smithi007.stdout: debug mds migrator = 0/0 2023-11-27T09:25:01.896 INFO:teuthology.orchestra.run.smithi007.stdout: debug memdb = 0/0 2023-11-27T09:25:01.897 INFO:teuthology.orchestra.run.smithi007.stdout: debug mgr = 0/0 2023-11-27T09:25:01.897 INFO:teuthology.orchestra.run.smithi007.stdout: debug mgrc = 0/0 2023-11-27T09:25:01.897 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 0/0 2023-11-27T09:25:01.897 INFO:teuthology.orchestra.run.smithi007.stdout: debug monc = 0/0 2023-11-27T09:25:01.897 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 0/0 2023-11-27T09:25:01.897 INFO:teuthology.orchestra.run.smithi007.stdout: debug objclass = 0/0 2023-11-27T09:25:01.897 INFO:teuthology.orchestra.run.smithi007.stdout: debug objectcacher = 0/0 2023-11-27T09:25:01.897 INFO:teuthology.orchestra.run.smithi007.stdout: debug objecter = 0/0 2023-11-27T09:25:01.897 INFO:teuthology.orchestra.run.smithi007.stdout: debug optracker = 0/0 2023-11-27T09:25:01.897 INFO:teuthology.orchestra.run.smithi007.stdout: debug osd = 0/0 2023-11-27T09:25:01.897 INFO:teuthology.orchestra.run.smithi007.stdout: debug paxos = 0/0 2023-11-27T09:25:01.897 INFO:teuthology.orchestra.run.smithi007.stdout: debug perfcounter = 0/0 2023-11-27T09:25:01.897 INFO:teuthology.orchestra.run.smithi007.stdout: debug rados = 0/0 2023-11-27T09:25:01.897 INFO:teuthology.orchestra.run.smithi007.stdout: debug rbd = 0/0 2023-11-27T09:25:01.898 INFO:teuthology.orchestra.run.smithi007.stdout: debug rbd mirror = 0/0 2023-11-27T09:25:01.898 INFO:teuthology.orchestra.run.smithi007.stdout: debug rbd replay = 0/0 2023-11-27T09:25:01.898 INFO:teuthology.orchestra.run.smithi007.stdout: debug refs = 0/0 2023-11-27T09:25:01.898 INFO:teuthology.orchestra.run.smithi007.stdout: debug rgw = 0/0 2023-11-27T09:25:01.898 INFO:teuthology.orchestra.run.smithi007.stdout: debug rgw sync = 0/0 2023-11-27T09:25:01.898 INFO:teuthology.orchestra.run.smithi007.stdout: debug rocksdb = 0/0 2023-11-27T09:25:01.898 INFO:teuthology.orchestra.run.smithi007.stdout: debug striper = 0/0 2023-11-27T09:25:01.898 INFO:teuthology.orchestra.run.smithi007.stdout: debug throttle = 0/0 2023-11-27T09:25:01.898 INFO:teuthology.orchestra.run.smithi007.stdout: debug timer = 0/0 2023-11-27T09:25:01.898 INFO:teuthology.orchestra.run.smithi007.stdout: debug tp = 0/0 2023-11-27T09:25:01.898 INFO:teuthology.orchestra.run.smithi007.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-11-27T09:25:01.898 INFO:teuthology.orchestra.run.smithi007.stdout: ms client mode = crc 2023-11-27T09:25:01.898 INFO:teuthology.orchestra.run.smithi007.stdout: ms cluster mode = crc 2023-11-27T09:25:01.898 INFO:teuthology.orchestra.run.smithi007.stdout: ms mon client mode = crc 2023-11-27T09:25:01.898 INFO:teuthology.orchestra.run.smithi007.stdout: ms mon cluster mode = crc 2023-11-27T09:25:01.899 INFO:teuthology.orchestra.run.smithi007.stdout: ms mon service mode = crc 2023-11-27T09:25:01.899 INFO:teuthology.orchestra.run.smithi007.stdout: ms service mode = crc 2023-11-27T09:25:01.899 INFO:teuthology.orchestra.run.smithi007.stdout: fsid = ffcd362b-f2f8-4720-b3d4-ddcd1c2d0a43 2023-11-27T09:25:01.899 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.899 INFO:teuthology.orchestra.run.smithi007.stdout:[osd] 2023-11-27T09:25:01.899 INFO:teuthology.orchestra.run.smithi007.stdout: osd journal size = 100 2023-11-27T09:25:01.899 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.899 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub load threshold = 5.0 2023-11-27T09:25:01.899 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub max interval = 600 2023-11-27T09:25:01.899 INFO:teuthology.orchestra.run.smithi007.stdout: osd mclock profile = high_recovery_ops 2023-11-27T09:25:01.899 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.899 INFO:teuthology.orchestra.run.smithi007.stdout: osd recover clone overlap = true 2023-11-27T09:25:01.899 INFO:teuthology.orchestra.run.smithi007.stdout: osd recovery max chunk = 1048576 2023-11-27T09:25:01.899 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.899 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug shutdown = true 2023-11-27T09:25:01.900 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug op order = true 2023-11-27T09:25:01.900 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify stray on activate = true 2023-11-27T09:25:01.900 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug trim objects = true 2023-11-27T09:25:01.900 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.900 INFO:teuthology.orchestra.run.smithi007.stdout: osd open classes on start = true 2023-11-27T09:25:01.900 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug pg log writeout = true 2023-11-27T09:25:01.900 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.900 INFO:teuthology.orchestra.run.smithi007.stdout: osd deep scrub update digest min age = 30 2023-11-27T09:25:01.900 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.900 INFO:teuthology.orchestra.run.smithi007.stdout: osd map max advance = 10 2023-11-27T09:25:01.900 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.900 INFO:teuthology.orchestra.run.smithi007.stdout: journal zero on create = true 2023-11-27T09:25:01.900 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.900 INFO:teuthology.orchestra.run.smithi007.stdout: filestore ondisk finisher threads = 3 2023-11-27T09:25:01.900 INFO:teuthology.orchestra.run.smithi007.stdout: filestore apply finisher threads = 3 2023-11-27T09:25:01.901 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.901 INFO:teuthology.orchestra.run.smithi007.stdout: bdev debug aio = true 2023-11-27T09:25:01.901 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug misdirected ops = true 2023-11-27T09:25:01.901 INFO:teuthology.orchestra.run.smithi007.stdout: debug monc = 20 2023-11-27T09:25:01.901 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore block size = 96636764160 2023-11-27T09:25:01.901 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore compression algorithm = snappy 2023-11-27T09:25:01.901 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore compression mode = aggressive 2023-11-27T09:25:01.901 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore fsck on mount = True 2023-11-27T09:25:01.901 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore rocksdb cf = False 2023-11-27T09:25:01.901 INFO:teuthology.orchestra.run.smithi007.stdout: crimson seastar smp = 3 2023-11-27T09:25:01.901 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluefs = 20 2023-11-27T09:25:01.901 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluestore = 20 2023-11-27T09:25:01.901 INFO:teuthology.orchestra.run.smithi007.stdout: debug filestore = 0/0 2023-11-27T09:25:01.901 INFO:teuthology.orchestra.run.smithi007.stdout: debug journal = 0/0 2023-11-27T09:25:01.902 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 0/0 2023-11-27T09:25:01.902 INFO:teuthology.orchestra.run.smithi007.stdout: debug osd = 0/0 2023-11-27T09:25:01.902 INFO:teuthology.orchestra.run.smithi007.stdout: debug rocksdb = 10 2023-11-27T09:25:01.902 INFO:teuthology.orchestra.run.smithi007.stdout: err to stderr = True 2023-11-27T09:25:01.902 INFO:teuthology.orchestra.run.smithi007.stdout: log flush on exit = True 2023-11-27T09:25:01.902 INFO:teuthology.orchestra.run.smithi007.stdout: log to file = False 2023-11-27T09:25:01.902 INFO:teuthology.orchestra.run.smithi007.stdout: log to stderr = True 2023-11-27T09:25:01.902 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd backfillfull_ratio = 0.85 2023-11-27T09:25:01.902 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd full ratio = 0.9 2023-11-27T09:25:01.902 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd nearfull ratio = 0.8 2023-11-27T09:25:01.902 INFO:teuthology.orchestra.run.smithi007.stdout: osd failsafe full ratio = 0.95 2023-11-27T09:25:01.902 INFO:teuthology.orchestra.run.smithi007.stdout: osd objectstore = bluestore 2023-11-27T09:25:01.902 INFO:teuthology.orchestra.run.smithi007.stdout: osd shutdown pgref assert = True 2023-11-27T09:25:01.902 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.902 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr] 2023-11-27T09:25:01.903 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2023-11-27T09:25:01.903 INFO:teuthology.orchestra.run.smithi007.stdout: debug mgr = 20 2023-11-27T09:25:01.903 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2023-11-27T09:25:01.903 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2023-11-27T09:25:01.903 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2023-11-27T09:25:01.903 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2023-11-27T09:25:01.903 INFO:teuthology.orchestra.run.smithi007.stdout: mgr/telemetry/nag = false 2023-11-27T09:25:01.903 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.903 INFO:teuthology.orchestra.run.smithi007.stdout:[mon] 2023-11-27T09:25:01.903 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 0/0 2023-11-27T09:25:01.903 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 0/0 2023-11-27T09:25:01.903 INFO:teuthology.orchestra.run.smithi007.stdout: debug paxos = 0/0 2023-11-27T09:25:01.903 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2023-11-27T09:25:01.903 INFO:teuthology.orchestra.run.smithi007.stdout: mon data avail warn = 5 2023-11-27T09:25:01.904 INFO:teuthology.orchestra.run.smithi007.stdout: mon mgr mkfs grace = 240 2023-11-27T09:25:01.904 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2023-11-27T09:25:01.904 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd reporter subtree level = osd 2023-11-27T09:25:01.904 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd prime pg temp = true 2023-11-27T09:25:01.904 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2023-11-27T09:25:01.904 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.904 INFO:teuthology.orchestra.run.smithi007.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-27T09:25:01.904 INFO:teuthology.orchestra.run.smithi007.stdout: auth mon ticket ttl = 660 # 11m 2023-11-27T09:25:01.904 INFO:teuthology.orchestra.run.smithi007.stdout: auth service ticket ttl = 240 # 4m 2023-11-27T09:25:01.904 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.904 INFO:teuthology.orchestra.run.smithi007.stdout: # don't complain about insecure global_id in the test suite 2023-11-27T09:25:01.904 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-27T09:25:01.904 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-27T09:25:01.904 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.904 INFO:teuthology.orchestra.run.smithi007.stdout: # 1m isn't quite enough 2023-11-27T09:25:01.904 INFO:teuthology.orchestra.run.smithi007.stdout: mon_down_mkfs_grace = 2m 2023-11-27T09:25:01.905 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.905 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_filestore_osds = false 2023-11-27T09:25:01.905 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default crimson = True 2023-11-27T09:25:01.905 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:01.905 INFO:teuthology.orchestra.run.smithi007.stdout:[client] 2023-11-27T09:25:01.905 INFO:teuthology.orchestra.run.smithi007.stdout: rgw cache enabled = true 2023-11-27T09:25:01.905 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable ops log = true 2023-11-27T09:25:01.905 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable usage log = true 2023-11-27T09:25:01.905 INFO:teuthology.orchestra.run.smithi007.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-27T09:25:01.905 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-27T09:25:01.905 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.a] 2023-11-27T09:25:01.908 INFO:tasks.ceph:Creating admin key on mon.a... 2023-11-27T09:25:01.908 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-11-27T09:25:01.974 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-27T09:25:01.974 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:25:01.974 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-11-27T09:25:01.989 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:25:01.989 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-11-27T09:25:02.044 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:25:02.045 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:25:02.045 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-27T09:25:02.045 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-27T09:25:02.131 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:25:02.131 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-27T09:25:02.187 INFO:tasks.ceph:Setting up mon nodes... 2023-11-27T09:25:02.187 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-27T09:25:02.187 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-11-27T09:25:02.294 INFO:teuthology.orchestra.run.smithi007.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-11-27T09:25:02.297 INFO:tasks.ceph:Setting up mds nodes... 2023-11-27T09:25:02.297 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-27T09:25:02.298 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-27T09:25:02.349 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-11-27T09:25:02.368 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-27T09:25:02.369 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-27T09:25:02.369 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-11-27T09:25:02.399 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-27T09:25:02.399 INFO:tasks.ceph:role: osd.0 2023-11-27T09:25:02.399 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:25:02.399 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-27T09:25:02.469 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-27T09:25:02.469 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-27T09:25:02.469 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-27T09:25:02.469 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-27T09:25:02.469 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-27T09:25:02.470 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2023-11-27T09:25:02.470 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-27T09:25:02.470 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-27T09:25:02.470 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-27T09:25:02.470 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-27T09:25:02.501 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2023-11-27T09:25:02.503 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2023-11-27T09:25:02.503 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-11-27T09:25:02.578 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-11-27T09:25:02.654 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-11-27T09:25:02.725 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-27T09:25:02.725 INFO:tasks.ceph:role: osd.1 2023-11-27T09:25:02.725 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:25:02.725 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-27T09:25:02.795 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-27T09:25:02.795 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-27T09:25:02.795 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-27T09:25:02.795 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-27T09:25:02.795 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-27T09:25:02.795 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2023-11-27T09:25:02.795 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-27T09:25:02.796 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-27T09:25:02.796 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-27T09:25:02.796 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-27T09:25:02.828 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2023-11-27T09:25:02.829 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2023-11-27T09:25:02.830 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-11-27T09:25:02.903 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-11-27T09:25:02.979 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-11-27T09:25:03.050 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-27T09:25:03.050 INFO:tasks.ceph:role: osd.2 2023-11-27T09:25:03.050 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:25:03.051 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-27T09:25:03.123 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-27T09:25:03.123 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-27T09:25:03.123 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-27T09:25:03.123 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-27T09:25:03.123 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-27T09:25:03.123 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2023-11-27T09:25:03.123 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-27T09:25:03.123 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-27T09:25:03.123 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-27T09:25:03.123 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-27T09:25:03.155 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2023-11-27T09:25:03.157 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2023-11-27T09:25:03.157 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-11-27T09:25:03.231 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-11-27T09:25:03.307 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-27T09:25:03.873 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-27T09:25:03.882 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:03,883 seastar - Reactor backend: linux-aio 2023-11-27T09:25:03.901 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:03,901 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-27T09:25:03.902 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:03,901 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-27T09:25:03.902 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:03,901 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-27T09:25:03.902 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:03,901 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-27T09:25:03.905 INFO:teuthology.orchestra.run.smithi007.stderr:==80857==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-27T09:25:03.911 INFO:teuthology.orchestra.run.smithi007.stderr:==80857==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd2e421000; bottom 0x7ff2b6e71000; size: 0x000a775b0000 (44952125440) 2023-11-27T09:25:03.911 INFO:teuthology.orchestra.run.smithi007.stderr:False positive error reports may follow 2023-11-27T09:25:03.911 INFO:teuthology.orchestra.run.smithi007.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-27T09:25:04.485 INFO:teuthology.orchestra.run.smithi007.stderr:----------------------------------------------------- 2023-11-27T09:25:04.485 INFO:teuthology.orchestra.run.smithi007.stderr:Suppressions used: 2023-11-27T09:25:04.485 INFO:teuthology.orchestra.run.smithi007.stderr: count bytes template 2023-11-27T09:25:04.485 INFO:teuthology.orchestra.run.smithi007.stderr: 1 8 MallocExtension::Register 2023-11-27T09:25:04.485 INFO:teuthology.orchestra.run.smithi007.stderr:----------------------------------------------------- 2023-11-27T09:25:04.485 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-27T09:25:04.540 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-27T09:25:04.541 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:04,541 seastar - Reactor backend: linux-aio 2023-11-27T09:25:04.559 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:04,559 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-27T09:25:04.559 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:04,559 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-27T09:25:04.559 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:04,559 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-27T09:25:04.566 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:04,567 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-27T09:25:04.569 INFO:teuthology.orchestra.run.smithi007.stderr:==80855==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-27T09:25:04.575 INFO:teuthology.orchestra.run.smithi007.stderr:==80855==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd2e421000; bottom 0x7ff2b4c6a000; size: 0x000a797b7000 (44987805696) 2023-11-27T09:25:04.575 INFO:teuthology.orchestra.run.smithi007.stderr:False positive error reports may follow 2023-11-27T09:25:04.575 INFO:teuthology.orchestra.run.smithi007.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-27T09:25:04.660 INFO:teuthology.orchestra.run.smithi007.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c8534d2 0x2c854880 0x2c855dce 0x2c856152 0x2c85629c 0x2c8566f2 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x263d7ce4 0x263da32d 0x263dc13e 0x263de9a3 0x2620f62d 0x1ed74304 0x1ece999b 0x1eced7dc 0x2cbd6bf5 2023-11-27T09:25:04.660 INFO:teuthology.orchestra.run.smithi007.stderr:kernel callstack: 2023-11-27T09:25:08.628 INFO:teuthology.orchestra.run.smithi007.stderr:----------------------------------------------------- 2023-11-27T09:25:08.628 INFO:teuthology.orchestra.run.smithi007.stderr:Suppressions used: 2023-11-27T09:25:08.628 INFO:teuthology.orchestra.run.smithi007.stderr: count bytes template 2023-11-27T09:25:08.628 INFO:teuthology.orchestra.run.smithi007.stderr: 1 8 MallocExtension::Register 2023-11-27T09:25:08.628 INFO:teuthology.orchestra.run.smithi007.stderr:----------------------------------------------------- 2023-11-27T09:25:08.629 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-27T09:25:08.629 INFO:teuthology.orchestra.run.smithi007.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid ffcd362b-f2f8-4720-b3d4-ddcd1c2d0a43 2023-11-27T09:25:08.666 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-11-27T09:25:08.697 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-27T09:25:09.249 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-27T09:25:09.249 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:09,250 seastar - Reactor backend: linux-aio 2023-11-27T09:25:09.270 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:09,269 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-27T09:25:09.270 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:09,269 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-27T09:25:09.270 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:09,269 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-27T09:25:09.270 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:09,269 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-27T09:25:09.272 INFO:teuthology.orchestra.run.smithi007.stderr:==81040==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-27T09:25:09.278 INFO:teuthology.orchestra.run.smithi007.stderr:==81040==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd4edaf000; bottom 0x7fc91f986000; size: 0x00342f429000 (224131190784) 2023-11-27T09:25:09.278 INFO:teuthology.orchestra.run.smithi007.stderr:False positive error reports may follow 2023-11-27T09:25:09.278 INFO:teuthology.orchestra.run.smithi007.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-27T09:25:09.304 INFO:teuthology.orchestra.run.smithi007.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c8534d2 0x2c854880 0x2c855dce 0x2c856152 0x2c85629c 0x2c8566f2 0x54daf 0x22dab314 0x22e3b14f 0x22e3fdd8 0x22e40326 0x22e35132 0x22e36ffb 0x22e4fba7 0x22e52b38 0x22e7784c 0x22e96ab1 0x22e97339 0x22e44f49 0x22e54569 0x22e55c65 0x22e5663e 0x22e57729 0x22f39638 0x22d999f9 0x22e685b4 0x22e69fff 0x22e6ad68 0x22e32bd4 0x22e3222c 0x22b43972 0x24b7ad80 0x24b82536 0x24b84e84 0x24b8605d 0x226e40fd 0x22734d5a 0x22758a8f 0x2c7f57ed 0x2c8935fa 0x2cab7898 0x2cab95bb 0x2c50edac 0x2c510cb5 0x1ee402fb 0x1ee4a469 0x1ecb9e70 0x3feaf 0x3ff5f 0x1e764a44 2023-11-27T09:25:09.304 INFO:teuthology.orchestra.run.smithi007.stderr:kernel callstack: 2023-11-27T09:25:09.827 INFO:teuthology.orchestra.run.smithi007.stderr:----------------------------------------------------- 2023-11-27T09:25:09.827 INFO:teuthology.orchestra.run.smithi007.stderr:Suppressions used: 2023-11-27T09:25:09.828 INFO:teuthology.orchestra.run.smithi007.stderr: count bytes template 2023-11-27T09:25:09.828 INFO:teuthology.orchestra.run.smithi007.stderr: 1 8 MallocExtension::Register 2023-11-27T09:25:09.828 INFO:teuthology.orchestra.run.smithi007.stderr:----------------------------------------------------- 2023-11-27T09:25:09.828 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-27T09:25:09.886 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-27T09:25:09.887 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:09,887 seastar - Reactor backend: linux-aio 2023-11-27T09:25:09.905 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:09,905 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-27T09:25:09.905 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:09,905 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-27T09:25:09.905 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:09,905 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-27T09:25:09.912 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:09,912 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-27T09:25:09.915 INFO:teuthology.orchestra.run.smithi007.stderr:==81038==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-27T09:25:09.921 INFO:teuthology.orchestra.run.smithi007.stderr:==81038==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd4edaf000; bottom 0x7fc91d77f000; size: 0x003431630000 (224166871040) 2023-11-27T09:25:09.921 INFO:teuthology.orchestra.run.smithi007.stderr:False positive error reports may follow 2023-11-27T09:25:09.921 INFO:teuthology.orchestra.run.smithi007.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-27T09:25:09.947 INFO:teuthology.orchestra.run.smithi007.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c8534d2 0x2c854880 0x2c855dce 0x2c856152 0x2c85629c 0x2c8566f2 0x54daf 0x1ebf48f1 0x22c98dd2 0x22c9e35e 0x24b7b84b 0x24b82536 0x24b84e84 0x24b8605d 0x226e40fd 0x22734d5a 0x22758a8f 0x2c7f57ed 0x2c8935fa 0x2cab7898 0x2cab95bb 0x2c50edac 0x2c510cb5 0x1ecbb726 0x3feaf 0x3ff5f 0x1e764a44 2023-11-27T09:25:09.947 INFO:teuthology.orchestra.run.smithi007.stderr:kernel callstack: 2023-11-27T09:25:13.964 INFO:teuthology.orchestra.run.smithi007.stderr:----------------------------------------------------- 2023-11-27T09:25:13.964 INFO:teuthology.orchestra.run.smithi007.stderr:Suppressions used: 2023-11-27T09:25:13.964 INFO:teuthology.orchestra.run.smithi007.stderr: count bytes template 2023-11-27T09:25:13.964 INFO:teuthology.orchestra.run.smithi007.stderr: 1 8 MallocExtension::Register 2023-11-27T09:25:13.964 INFO:teuthology.orchestra.run.smithi007.stderr:----------------------------------------------------- 2023-11-27T09:25:13.964 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-27T09:25:13.964 INFO:teuthology.orchestra.run.smithi007.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid ffcd362b-f2f8-4720-b3d4-ddcd1c2d0a43 2023-11-27T09:25:14.002 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-11-27T09:25:14.033 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-27T09:25:14.588 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-27T09:25:14.588 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:14,589 seastar - Reactor backend: linux-aio 2023-11-27T09:25:14.608 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:14,608 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-27T09:25:14.608 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:14,608 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-27T09:25:14.609 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:14,608 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-27T09:25:14.609 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:14,608 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-27T09:25:14.610 INFO:teuthology.orchestra.run.smithi007.stderr:==81222==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-27T09:25:14.617 INFO:teuthology.orchestra.run.smithi007.stderr:==81222==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffdddb69000; bottom 0x7f05bbdb6000; size: 0x00f821db3000 (1065719902208) 2023-11-27T09:25:14.617 INFO:teuthology.orchestra.run.smithi007.stderr:False positive error reports may follow 2023-11-27T09:25:14.617 INFO:teuthology.orchestra.run.smithi007.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-27T09:25:14.643 INFO:teuthology.orchestra.run.smithi007.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c8534d2 0x2c854880 0x2c855dce 0x2c856152 0x2c85629c 0x2c8566f2 0x54daf 0x22e382ed 0x22e38e90 0x22e7c131 0x22e84380 0x22e85544 0x22e55440 0x22e5663e 0x22e57729 0x22f39638 0x22d999f9 0x22e685b4 0x22e69fff 0x22e6ad68 0x22e32bd4 0x22e3222c 0x22b43972 0x24b7ad80 0x24b82536 0x24b84e84 0x24b8605d 0x226e40fd 0x22734d5a 0x22758a8f 0x2c7f57ed 0x2c8935fa 0x2cab7898 0x2cab95bb 0x2c50edac 0x2c510cb5 0x1ee402fb 0x1ee4a469 0x1ecb9e70 0x3feaf 0x3ff5f 0x1e764a44 2023-11-27T09:25:14.643 INFO:teuthology.orchestra.run.smithi007.stderr:kernel callstack: 2023-11-27T09:25:15.161 INFO:teuthology.orchestra.run.smithi007.stderr:----------------------------------------------------- 2023-11-27T09:25:15.162 INFO:teuthology.orchestra.run.smithi007.stderr:Suppressions used: 2023-11-27T09:25:15.162 INFO:teuthology.orchestra.run.smithi007.stderr: count bytes template 2023-11-27T09:25:15.162 INFO:teuthology.orchestra.run.smithi007.stderr: 1 8 MallocExtension::Register 2023-11-27T09:25:15.162 INFO:teuthology.orchestra.run.smithi007.stderr:----------------------------------------------------- 2023-11-27T09:25:15.162 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-27T09:25:15.217 INFO:teuthology.orchestra.run.smithi007.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-27T09:25:15.218 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:15,218 seastar - Reactor backend: linux-aio 2023-11-27T09:25:15.245 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:15,245 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-27T09:25:15.246 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:15,245 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-27T09:25:15.246 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:15,245 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-27T09:25:15.253 INFO:teuthology.orchestra.run.smithi007.stderr:INFO 2023-11-27 09:25:15,253 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-27T09:25:15.255 INFO:teuthology.orchestra.run.smithi007.stderr:==81220==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-27T09:25:15.261 INFO:teuthology.orchestra.run.smithi007.stderr:==81220==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffdddb69000; bottom 0x7f05b9baf000; size: 0x00f823fba000 (1065755582464) 2023-11-27T09:25:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:False positive error reports may follow 2023-11-27T09:25:15.262 INFO:teuthology.orchestra.run.smithi007.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-27T09:25:15.287 INFO:teuthology.orchestra.run.smithi007.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c8534d2 0x2c854880 0x2c855dce 0x2c856152 0x2c85629c 0x2c8566f2 0x54daf 0x1e91200d 0x22c99648 0x22c9e35e 0x24b7b84b 0x24b82536 0x24b84e84 0x24b8605d 0x226e40fd 0x22734d5a 0x22758a8f 0x2c7f57ed 0x2c8935fa 0x2cab7898 0x2cab95bb 0x2c50edac 0x2c510cb5 0x1ecbb726 0x3feaf 0x3ff5f 0x1e764a44 2023-11-27T09:25:15.287 INFO:teuthology.orchestra.run.smithi007.stderr:kernel callstack: 2023-11-27T09:25:19.304 INFO:teuthology.orchestra.run.smithi007.stderr:----------------------------------------------------- 2023-11-27T09:25:19.304 INFO:teuthology.orchestra.run.smithi007.stderr:Suppressions used: 2023-11-27T09:25:19.304 INFO:teuthology.orchestra.run.smithi007.stderr: count bytes template 2023-11-27T09:25:19.304 INFO:teuthology.orchestra.run.smithi007.stderr: 1 8 MallocExtension::Register 2023-11-27T09:25:19.304 INFO:teuthology.orchestra.run.smithi007.stderr:----------------------------------------------------- 2023-11-27T09:25:19.304 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-27T09:25:19.304 INFO:teuthology.orchestra.run.smithi007.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid ffcd362b-f2f8-4720-b3d4-ddcd1c2d0a43 2023-11-27T09:25:19.343 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-11-27T09:25:19.374 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-27T09:25:19.374 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:25:19.375 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-11-27T09:25:19.445 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:25:19.446 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-11-27T09:25:19.516 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:25:19.516 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-11-27T09:25:19.587 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:25:19.587 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-11-27T09:25:19.658 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:25:19.658 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-11-27T09:25:19.714 INFO:tasks.ceph:Adding keys to all mons... 2023-11-27T09:25:19.714 DEBUG:teuthology.orchestra.run.smithi007:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-27T09:25:19.782 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr.x] 2023-11-27T09:25:19.783 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBuYGRll32aERAAwqpLzL3nzJ845Yij724nLQ== 2023-11-27T09:25:19.783 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.0] 2023-11-27T09:25:19.783 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBwYGRl5+NeJRAAgHnwHGA5N55v5k6cWvAYWw== 2023-11-27T09:25:19.783 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.1] 2023-11-27T09:25:19.783 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQB1YGRlfuPaORAAfZWjhgHzuXeNCAFJj1FL7Q== 2023-11-27T09:25:19.783 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.2] 2023-11-27T09:25:19.783 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQB7YGRlZGKQEhAAivErV6Ok/A4s4y/4srZqlA== 2023-11-27T09:25:19.783 INFO:teuthology.orchestra.run.smithi007.stdout:[client.0] 2023-11-27T09:25:19.783 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQBuYGRljI3fFBAAA3M8nvIjf/ec8MKIWIunAQ== 2023-11-27T09:25:19.784 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-27T09:25:19.886 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-27T09:25:19.948 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-27T09:25:20.009 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-27T09:25:20.072 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-27T09:25:20.136 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-27T09:25:20.136 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-11-27T09:25:20.167 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-27T09:25:20.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:20.270+0000 7efc62c14980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:20.271 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:20.270+0000 7efc62c14980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:20.286 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-11-27T09:25:20.317 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-27T09:25:20.372 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-11-27T09:25:20.373 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-27T09:25:20.373 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-11-27T09:25:20.415 INFO:tasks.ceph.mon.a:Started 2023-11-27T09:25:20.415 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-11-27T09:25:20.415 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-11-27T09:25:20.415 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-11-27T09:25:20.418 INFO:tasks.ceph.mgr.x:Started 2023-11-27T09:25:20.418 INFO:tasks.ceph:Setting crush tunables to default 2023-11-27T09:25:20.418 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd crush tunables default 2023-11-27T09:25:20.501 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-11-27T09:25:20.500+0000 7fec4c25a980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:20.501 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-11-27T09:25:20.500+0000 7fec4c25a980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:20.501 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-11-27T09:25:20.500+0000 7fec4c25a980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:20.502 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:20.501+0000 7f0d83169f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:20.540 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:20.534+0000 7f503246d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:20.560 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:20.559+0000 7f503246d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:20.630 INFO:teuthology.orchestra.run.smithi007.stderr:adjusted tunables profile to default 2023-11-27T09:25:20.642 INFO:tasks.ceph:check_enable_crimson: True 2023-11-27T09:25:20.642 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2023-11-27T09:25:20.642 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2023-11-27T09:25:20.705 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:20.704+0000 7f0d83169f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:20.705 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:20.704+0000 7f0d83169f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:20.753 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:20.748+0000 7fa4c2d15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:20.755 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:20.754+0000 7fa4c2d15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:20.836 INFO:tasks.ceph.mon.a.smithi007.stderr:2023-11-27T09:25:20.835+0000 7fec441f1640 -1 WARNING: experimental feature 'crimson' is enabled 2023-11-27T09:25:20.836 INFO:tasks.ceph.mon.a.smithi007.stderr:Please be aware that this feature is experimental, untested, 2023-11-27T09:25:20.836 INFO:tasks.ceph.mon.a.smithi007.stderr:unsupported, and may result in data corruption, data loss, 2023-11-27T09:25:20.836 INFO:tasks.ceph.mon.a.smithi007.stderr:and/or irreparable damage to your cluster. Do not use 2023-11-27T09:25:20.836 INFO:tasks.ceph.mon.a.smithi007.stderr:feature with important data. 2023-11-27T09:25:20.837 INFO:tasks.ceph.mon.a.smithi007.stderr: 2023-11-27T09:25:21.198 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:21.197+0000 7f0d83169f00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-27T09:25:21.252 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:21.251+0000 7f0d83169f00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-27T09:25:21.643 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-11-27T09:25:21.643 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:25:21.644 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-11-27T09:25:21.675 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:25:21.675 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-11-27T09:25:21.745 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:25:21.746 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-11-27T09:25:21.796 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:21.795+0000 7f0d83169f00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-27T09:25:21.816 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd new d8d8c11b-4cd5-479a-bff1-2ca9c54cef41 0 2023-11-27T09:25:21.852 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:21.851+0000 7f0d83169f00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-27T09:25:21.900 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:21.899+0000 7f0d83169f00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-27T09:25:21.959 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:21.958+0000 7f0d83169f00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-27T09:25:21.964 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:21.959+0000 7f7e7de2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:21.966 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:21.966+0000 7f7e7de2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:22.004 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:22.004+0000 7f0d83169f00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-27T09:25:22.046 INFO:teuthology.orchestra.run.smithi007.stdout:0 2023-11-27T09:25:22.058 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd new c483f98f-5dab-4990-8ffa-725e28d1167d 1 2023-11-27T09:25:22.113 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:22.112+0000 7f0d83169f00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-27T09:25:22.168 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:22.163+0000 7f527591e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:22.170 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:22.170+0000 7f527591e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:22.190 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:22.189+0000 7f0d83169f00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-27T09:25:22.239 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:22.238+0000 7f0d83169f00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-27T09:25:22.249 INFO:teuthology.orchestra.run.smithi007.stdout:1 2023-11-27T09:25:22.260 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd new eac4f75e-4af5-4d58-b547-059ef92056ae 2 2023-11-27T09:25:22.295 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:22.295+0000 7f0d83169f00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-27T09:25:22.347 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:22.346+0000 7f0d83169f00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-27T09:25:22.371 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:22.366+0000 7fe77c19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:22.373 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:22.373+0000 7fe77c19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:22.452 INFO:teuthology.orchestra.run.smithi007.stdout:2 2023-11-27T09:25:22.463 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2023-11-27T09:25:22.538 INFO:tasks.ceph.osd.0:Restarting daemon 2023-11-27T09:25:22.539 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2023-11-27T09:25:22.581 INFO:tasks.ceph.osd.0:Started 2023-11-27T09:25:22.581 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2023-11-27T09:25:22.617 INFO:tasks.ceph.osd.1:Restarting daemon 2023-11-27T09:25:22.617 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2023-11-27T09:25:22.659 INFO:tasks.ceph.osd.1:Started 2023-11-27T09:25:22.659 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2023-11-27T09:25:22.699 INFO:tasks.ceph.osd.2:Restarting daemon 2023-11-27T09:25:22.699 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2023-11-27T09:25:22.742 INFO:tasks.ceph.osd.2:Started 2023-11-27T09:25:22.743 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-27T09:25:22.772 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:22.771+0000 7f0d83169f00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-27T09:25:22.878 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:22.873+0000 7f9e58c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:22.881 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:22.880+0000 7f9e58c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:22.936 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:22.935+0000 7f0d83169f00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-27T09:25:22.953 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:22.953 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":6,"fsid":"ffcd362b-f2f8-4720-b3d4-ddcd1c2d0a43","created":"2023-11-27T09:25:20.552090+0000","modified":"2023-11-27T09:25:22.451615+0000","last_up_change":"0.000000","last_in_change":"2023-11-27T09:25:22.451615+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"d8d8c11b-4cd5-479a-bff1-2ca9c54cef41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c483f98f-5dab-4990-8ffa-725e28d1167d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eac4f75e-4af5-4d58-b547-059ef92056ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-27T09:25:22.970 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-11-27T09:25:22.970 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-27T09:25:23.076 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:23.075+0000 7f0d83169f00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-27T09:25:23.134 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:23.134+0000 7f0d83169f00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-27T09:25:23.253 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:23.253+0000 7f0d83169f00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-27T09:25:23.273 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-27T09:25:23.387 INFO:teuthology.misc.health.smithi007.stderr:2023-11-27T09:25:23.383+0000 7fb2b642d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:23.390 INFO:teuthology.misc.health.smithi007.stderr:2023-11-27T09:25:23.389+0000 7fb2b642d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:23.400 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:23.400+0000 7f0d83169f00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-27T09:25:23.516 INFO:teuthology.misc.health.smithi007.stdout: 2023-11-27T09:25:23.516 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":6,"fsid":"ffcd362b-f2f8-4720-b3d4-ddcd1c2d0a43","created":"2023-11-27T09:25:20.552090+0000","modified":"2023-11-27T09:25:22.451615+0000","last_up_change":"0.000000","last_in_change":"2023-11-27T09:25:22.451615+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"d8d8c11b-4cd5-479a-bff1-2ca9c54cef41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c483f98f-5dab-4990-8ffa-725e28d1167d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"eac4f75e-4af5-4d58-b547-059ef92056ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-27T09:25:23.532 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-11-27T09:25:23.552 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:23.552+0000 7f0d83169f00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-27T09:25:23.600 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:23.599+0000 7f0d83169f00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-27T09:25:23.757 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:23.756+0000 7f0d83169f00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-27T09:25:24.622 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:24.621+0000 7f0d83169f00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-27T09:25:24.909 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:24.908+0000 7f0d83169f00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-27T09:25:24.964 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:24.963+0000 7f0d83169f00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-27T09:25:25.072 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:25.071+0000 7f0d83169f00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-27T09:25:26.091 INFO:tasks.ceph.mgr.x.smithi007.stderr:2023-11-27T09:25:26.090+0000 7f0d22d3d640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-11-27T09:25:29.833 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-27T09:25:29.943 INFO:teuthology.misc.health.smithi007.stderr:2023-11-27T09:25:29.940+0000 7fa46091e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:29.945 INFO:teuthology.misc.health.smithi007.stderr:2023-11-27T09:25:29.945+0000 7fa46091e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:30.131 INFO:teuthology.misc.health.smithi007.stdout: 2023-11-27T09:25:30.131 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":9,"fsid":"ffcd362b-f2f8-4720-b3d4-ddcd1c2d0a43","created":"2023-11-27T09:25:20.552090+0000","modified":"2023-11-27T09:25:30.098531+0000","last_up_change":"2023-11-27T09:25:28.092532+0000","last_in_change":"2023-11-27T09:25:22.451615+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-27T09:25:29.142764+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d8d8c11b-4cd5-479a-bff1-2ca9c54cef41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6805","nonce":2835085732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2835085732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2835085732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6809","nonce":2835085732}]},"public_addr":"172.21.15.7:6805/2835085732","cluster_addr":"172.21.15.7:6804/2835085732","heartbeat_back_addr":"172.21.15.7:6808/2835085732","heartbeat_front_addr":"172.21.15.7:6809/2835085732","state":["exists","up"]},{"osd":1,"uuid":"c483f98f-5dab-4990-8ffa-725e28d1167d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6803","nonce":1902004532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":1902004532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":1902004532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1902004532}]},"public_addr":"172.21.15.7:6803/1902004532","cluster_addr":"172.21.15.7:6802/1902004532","heartbeat_back_addr":"172.21.15.7:6811/1902004532","heartbeat_front_addr":"172.21.15.7:6810/1902004532","state":["exists","up"]},{"osd":2,"uuid":"eac4f75e-4af5-4d58-b547-059ef92056ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":1298969470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6807","nonce":1298969470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6813","nonce":1298969470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1298969470}]},"public_addr":"172.21.15.7:6806/1298969470","cluster_addr":"172.21.15.7:6807/1298969470","heartbeat_back_addr":"172.21.15.7:6813/1298969470","heartbeat_front_addr":"172.21.15.7:6812/1298969470","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-27T09:25:30.142 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-11-27T09:25:30.142 INFO:tasks.ceph:Creating RBD pool 2023-11-27T09:25:30.142 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd pool create rbd 8 2023-11-27T09:25:30.253 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:30.246+0000 7f9317e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:30.255 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:30.254+0000 7f9317e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:31.104 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'rbd' created 2023-11-27T09:25:31.120 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-11-27T09:25:31.249 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:31.239+0000 7f42e6c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:31.252 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:31.250+0000 7f42e6c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:32.108 INFO:teuthology.orchestra.run.smithi007.stderr:enabled application 'rbd' on pool 'rbd' 2023-11-27T09:25:32.123 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-11-27T09:25:32.123 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-27T09:25:32.124 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-11-27T09:25:32.124 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-11-27T09:25:32.124 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-27T09:25:32.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:32.240+0000 7f092791e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:32.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:32.246+0000 7f092791e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:32.483 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:32.493 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":3630021642},{"type":"v1","addr":"172.21.15.7:6801","nonce":3630021642}]},"active_addr":"172.21.15.7:6801/3630021642","active_change":"2023-11-27T09:25:25.084687+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":3081936191}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":3278068516}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":588048903}]}]} 2023-11-27T09:25:32.495 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-11-27T09:25:32.496 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-11-27T09:25:32.496 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-27T09:25:32.608 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:32.604+0000 7f430100f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:32.610 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:32.608+0000 7f430100f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:32.798 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:32.798 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":11,"fsid":"ffcd362b-f2f8-4720-b3d4-ddcd1c2d0a43","created":"2023-11-27T09:25:20.552090+0000","modified":"2023-11-27T09:25:32.105203+0000","last_up_change":"2023-11-27T09:25:28.092532+0000","last_in_change":"2023-11-27T09:25:22.451615+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-27T09:25:29.142764+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-27T09:25:30.468575+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d8d8c11b-4cd5-479a-bff1-2ca9c54cef41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6805","nonce":2835085732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2835085732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2835085732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6809","nonce":2835085732}]},"public_addr":"172.21.15.7:6805/2835085732","cluster_addr":"172.21.15.7:6804/2835085732","heartbeat_back_addr":"172.21.15.7:6808/2835085732","heartbeat_front_addr":"172.21.15.7:6809/2835085732","state":["exists","up"]},{"osd":1,"uuid":"c483f98f-5dab-4990-8ffa-725e28d1167d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6803","nonce":1902004532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":1902004532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":1902004532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1902004532}]},"public_addr":"172.21.15.7:6803/1902004532","cluster_addr":"172.21.15.7:6802/1902004532","heartbeat_back_addr":"172.21.15.7:6811/1902004532","heartbeat_front_addr":"172.21.15.7:6810/1902004532","state":["exists","up"]},{"osd":2,"uuid":"eac4f75e-4af5-4d58-b547-059ef92056ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":1298969470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6807","nonce":1298969470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6813","nonce":1298969470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1298969470}]},"public_addr":"172.21.15.7:6806/1298969470","cluster_addr":"172.21.15.7:6807/1298969470","heartbeat_back_addr":"172.21.15.7:6813/1298969470","heartbeat_front_addr":"172.21.15.7:6812/1298969470","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-27T09:25:32.810 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-11-27T09:25:32.811 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-27T09:25:32.923 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:32.919+0000 7f5555115640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:32.925 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:32.923+0000 7f5555115640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:33.108 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:33.108 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":11,"fsid":"ffcd362b-f2f8-4720-b3d4-ddcd1c2d0a43","created":"2023-11-27T09:25:20.552090+0000","modified":"2023-11-27T09:25:32.105203+0000","last_up_change":"2023-11-27T09:25:28.092532+0000","last_in_change":"2023-11-27T09:25:22.451615+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-27T09:25:29.142764+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-27T09:25:30.468575+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d8d8c11b-4cd5-479a-bff1-2ca9c54cef41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6805","nonce":2835085732}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2835085732}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2835085732}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6809","nonce":2835085732}]},"public_addr":"172.21.15.7:6805/2835085732","cluster_addr":"172.21.15.7:6804/2835085732","heartbeat_back_addr":"172.21.15.7:6808/2835085732","heartbeat_front_addr":"172.21.15.7:6809/2835085732","state":["exists","up"]},{"osd":1,"uuid":"c483f98f-5dab-4990-8ffa-725e28d1167d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6803","nonce":1902004532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":1902004532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6811","nonce":1902004532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1902004532}]},"public_addr":"172.21.15.7:6803/1902004532","cluster_addr":"172.21.15.7:6802/1902004532","heartbeat_back_addr":"172.21.15.7:6811/1902004532","heartbeat_front_addr":"172.21.15.7:6810/1902004532","state":["exists","up"]},{"osd":2,"uuid":"eac4f75e-4af5-4d58-b547-059ef92056ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":1298969470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6807","nonce":1298969470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6813","nonce":1298969470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1298969470}]},"public_addr":"172.21.15.7:6806/1298969470","cluster_addr":"172.21.15.7:6807/1298969470","heartbeat_back_addr":"172.21.15.7:6813/1298969470","heartbeat_front_addr":"172.21.15.7:6812/1298969470","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-27T09:25:33.122 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-11-27T09:25:33.122 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-11-27T09:25:33.123 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-11-27T09:25:33.244 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:33.239+0000 7f960742d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:33.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:33.239+0000 7f652399e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:33.245 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:33.239+0000 7f5a4775e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:33.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:33.245+0000 7f5a4775e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:33.247 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:33.246+0000 7f652399e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:33.248 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:33.246+0000 7f960742d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:33.275 INFO:teuthology.orchestra.run.smithi007.stdout:34359738373 2023-11-27T09:25:33.275 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-27T09:25:33.276 INFO:teuthology.orchestra.run.smithi007.stdout:34359738373 2023-11-27T09:25:33.277 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-27T09:25:33.277 INFO:teuthology.orchestra.run.smithi007.stdout:34359738373 2023-11-27T09:25:33.278 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-27T09:25:33.435 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:33.430+0000 7fba0722d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:33.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:33.430+0000 7f5b2726d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:33.436 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:33.430+0000 7fa063260640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:33.437 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:33.436+0000 7fba0722d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:33.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:33.436+0000 7f5b2726d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:33.438 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:33.436+0000 7fa063260640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:33.632 INFO:teuthology.orchestra.run.smithi007.stdout:34359738373 2023-11-27T09:25:33.639 INFO:teuthology.orchestra.run.smithi007.stdout:34359738373 2023-11-27T09:25:33.644 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2023-11-27T09:25:33.644 DEBUG:teuthology.parallel:result is None 2023-11-27T09:25:33.649 INFO:teuthology.orchestra.run.smithi007.stdout:34359738373 2023-11-27T09:25:33.652 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2023-11-27T09:25:33.652 DEBUG:teuthology.parallel:result is None 2023-11-27T09:25:33.662 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2023-11-27T09:25:33.662 DEBUG:teuthology.parallel:result is None 2023-11-27T09:25:33.662 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-27T09:25:33.662 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-27T09:25:33.813 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:33.808+0000 7f6de491e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:33.815 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:33.813+0000 7f6de491e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:33.997 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:33.998 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-11-27T09:25:34.009 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-11-27T09:25:33.264880+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80764,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034756,"statfs":{"total":289910292480,"available":289827590144,"internally_reserved":0,"allocated":274432,"data_stored":1227102,"data_compressed":59828,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-27T09:25:33.164857+0000","last_change":"2023-11-27T09:25:32.199595+0000","last_active":"2023-11-27T09:25:33.164857+0000","last_peered":"2023-11-27T09:25:33.164857+0000","last_clean":"2023-11-27T09:25:33.164857+0000","last_became_active":"2023-11-27T09:25:32.197775+0000","last_became_peered":"2023-11-27T09:25:32.197775+0000","last_unstale":"2023-11-27T09:25:33.164857+0000","last_undegraded":"2023-11-27T09:25:33.164857+0000","last_fullsized":"2023-11-27T09:25:33.164857+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_clean_scrub_stamp":"2023-11-27T09:25:31.101787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-27T09:25:33.162853+0000","last_change":"2023-11-27T09:25:32.217162+0000","last_active":"2023-11-27T09:25:33.162853+0000","last_peered":"2023-11-27T09:25:33.162853+0000","last_clean":"2023-11-27T09:25:33.162853+0000","last_became_active":"2023-11-27T09:25:32.209443+0000","last_became_peered":"2023-11-27T09:25:32.209443+0000","last_unstale":"2023-11-27T09:25:33.162853+0000","last_undegraded":"2023-11-27T09:25:33.162853+0000","last_fullsized":"2023-11-27T09:25:33.162853+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_clean_scrub_stamp":"2023-11-27T09:25:31.101787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-27T09:25:33.167686+0000","last_change":"2023-11-27T09:25:32.228525+0000","last_active":"2023-11-27T09:25:33.167686+0000","last_peered":"2023-11-27T09:25:33.167686+0000","last_clean":"2023-11-27T09:25:33.167686+0000","last_became_active":"2023-11-27T09:25:32.226300+0000","last_became_peered":"2023-11-27T09:25:32.226300+0000","last_unstale":"2023-11-27T09:25:33.167686+0000","last_undegraded":"2023-11-27T09:25:33.167686+0000","last_fullsized":"2023-11-27T09:25:33.167686+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_clean_scrub_stamp":"2023-11-27T09:25:31.101787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-27T09:25:33.148390+0000","last_change":"2023-11-27T09:25:32.213774+0000","last_active":"2023-11-27T09:25:33.148390+0000","last_peered":"2023-11-27T09:25:33.148390+0000","last_clean":"2023-11-27T09:25:33.148390+0000","last_became_active":"2023-11-27T09:25:32.211985+0000","last_became_peered":"2023-11-27T09:25:32.211985+0000","last_unstale":"2023-11-27T09:25:33.148390+0000","last_undegraded":"2023-11-27T09:25:33.148390+0000","last_fullsized":"2023-11-27T09:25:33.148390+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_clean_scrub_stamp":"2023-11-27T09:25:31.101787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-27T09:25:33.163628+0000","last_change":"2023-11-27T09:25:32.194598+0000","last_active":"2023-11-27T09:25:33.163628+0000","last_peered":"2023-11-27T09:25:33.163628+0000","last_clean":"2023-11-27T09:25:33.163628+0000","last_became_active":"2023-11-27T09:25:32.192319+0000","last_became_peered":"2023-11-27T09:25:32.192319+0000","last_unstale":"2023-11-27T09:25:33.163628+0000","last_undegraded":"2023-11-27T09:25:33.163628+0000","last_fullsized":"2023-11-27T09:25:33.163628+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_clean_scrub_stamp":"2023-11-27T09:25:31.101787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-27T09:25:33.117330+0000","last_change":"2023-11-27T09:25:32.193135+0000","last_active":"2023-11-27T09:25:33.117330+0000","last_peered":"2023-11-27T09:25:33.117330+0000","last_clean":"2023-11-27T09:25:33.117330+0000","last_became_active":"2023-11-27T09:25:32.188083+0000","last_became_peered":"2023-11-27T09:25:32.188083+0000","last_unstale":"2023-11-27T09:25:33.117330+0000","last_undegraded":"2023-11-27T09:25:33.117330+0000","last_fullsized":"2023-11-27T09:25:33.117330+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_clean_scrub_stamp":"2023-11-27T09:25:31.101787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-27T09:25:33.116739+0000","last_change":"2023-11-27T09:25:32.188002+0000","last_active":"2023-11-27T09:25:33.116739+0000","last_peered":"2023-11-27T09:25:33.116739+0000","last_clean":"2023-11-27T09:25:33.116739+0000","last_became_active":"2023-11-27T09:25:32.185114+0000","last_became_peered":"2023-11-27T09:25:32.185114+0000","last_unstale":"2023-11-27T09:25:33.116739+0000","last_undegraded":"2023-11-27T09:25:33.116739+0000","last_fullsized":"2023-11-27T09:25:33.116739+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_clean_scrub_stamp":"2023-11-27T09:25:31.101787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-27T09:25:33.165659+0000","last_change":"2023-11-27T09:25:32.178253+0000","last_active":"2023-11-27T09:25:33.165659+0000","last_peered":"2023-11-27T09:25:33.165659+0000","last_clean":"2023-11-27T09:25:33.165659+0000","last_became_active":"2023-11-27T09:25:32.176381+0000","last_became_peered":"2023-11-27T09:25:32.176381+0000","last_unstale":"2023-11-27T09:25:33.165659+0000","last_undegraded":"2023-11-27T09:25:33.165659+0000","last_fullsized":"2023-11-27T09:25:33.165659+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_clean_scrub_stamp":"2023-11-27T09:25:31.101787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-27T09:25:33.163465+0000","last_change":"2023-11-27T09:25:31.173601+0000","last_active":"2023-11-27T09:25:33.163465+0000","last_peered":"2023-11-27T09:25:33.163465+0000","last_clean":"2023-11-27T09:25:33.163465+0000","last_became_active":"2023-11-27T09:25:31.170067+0000","last_became_peered":"2023-11-27T09:25:31.170067+0000","last_unstale":"2023-11-27T09:25:33.163465+0000","last_undegraded":"2023-11-27T09:25:33.163465+0000","last_fullsized":"2023-11-27T09:25:33.163465+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-27T09:25:30.098531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-27T09:25:30.098531+0000","last_clean_scrub_stamp":"2023-11-27T09:25:30.098531+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":16996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":29914,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":29914,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-27T09:25:34.009 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-27T09:25:34.123 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:34.118+0000 7fca37aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:34.125 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:34.123+0000 7fca37aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:34.307 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:34.308 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2023-11-27T09:25:34.319 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-11-27T09:25:33.264880+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80764,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034756,"statfs":{"total":289910292480,"available":289827590144,"internally_reserved":0,"allocated":274432,"data_stored":1227102,"data_compressed":59828,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-27T09:25:33.164857+0000","last_change":"2023-11-27T09:25:32.199595+0000","last_active":"2023-11-27T09:25:33.164857+0000","last_peered":"2023-11-27T09:25:33.164857+0000","last_clean":"2023-11-27T09:25:33.164857+0000","last_became_active":"2023-11-27T09:25:32.197775+0000","last_became_peered":"2023-11-27T09:25:32.197775+0000","last_unstale":"2023-11-27T09:25:33.164857+0000","last_undegraded":"2023-11-27T09:25:33.164857+0000","last_fullsized":"2023-11-27T09:25:33.164857+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_clean_scrub_stamp":"2023-11-27T09:25:31.101787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-27T09:25:33.162853+0000","last_change":"2023-11-27T09:25:32.217162+0000","last_active":"2023-11-27T09:25:33.162853+0000","last_peered":"2023-11-27T09:25:33.162853+0000","last_clean":"2023-11-27T09:25:33.162853+0000","last_became_active":"2023-11-27T09:25:32.209443+0000","last_became_peered":"2023-11-27T09:25:32.209443+0000","last_unstale":"2023-11-27T09:25:33.162853+0000","last_undegraded":"2023-11-27T09:25:33.162853+0000","last_fullsized":"2023-11-27T09:25:33.162853+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_clean_scrub_stamp":"2023-11-27T09:25:31.101787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-27T09:25:33.167686+0000","last_change":"2023-11-27T09:25:32.228525+0000","last_active":"2023-11-27T09:25:33.167686+0000","last_peered":"2023-11-27T09:25:33.167686+0000","last_clean":"2023-11-27T09:25:33.167686+0000","last_became_active":"2023-11-27T09:25:32.226300+0000","last_became_peered":"2023-11-27T09:25:32.226300+0000","last_unstale":"2023-11-27T09:25:33.167686+0000","last_undegraded":"2023-11-27T09:25:33.167686+0000","last_fullsized":"2023-11-27T09:25:33.167686+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_clean_scrub_stamp":"2023-11-27T09:25:31.101787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-27T09:25:33.148390+0000","last_change":"2023-11-27T09:25:32.213774+0000","last_active":"2023-11-27T09:25:33.148390+0000","last_peered":"2023-11-27T09:25:33.148390+0000","last_clean":"2023-11-27T09:25:33.148390+0000","last_became_active":"2023-11-27T09:25:32.211985+0000","last_became_peered":"2023-11-27T09:25:32.211985+0000","last_unstale":"2023-11-27T09:25:33.148390+0000","last_undegraded":"2023-11-27T09:25:33.148390+0000","last_fullsized":"2023-11-27T09:25:33.148390+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_clean_scrub_stamp":"2023-11-27T09:25:31.101787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-27T09:25:33.163628+0000","last_change":"2023-11-27T09:25:32.194598+0000","last_active":"2023-11-27T09:25:33.163628+0000","last_peered":"2023-11-27T09:25:33.163628+0000","last_clean":"2023-11-27T09:25:33.163628+0000","last_became_active":"2023-11-27T09:25:32.192319+0000","last_became_peered":"2023-11-27T09:25:32.192319+0000","last_unstale":"2023-11-27T09:25:33.163628+0000","last_undegraded":"2023-11-27T09:25:33.163628+0000","last_fullsized":"2023-11-27T09:25:33.163628+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_clean_scrub_stamp":"2023-11-27T09:25:31.101787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-27T09:25:33.117330+0000","last_change":"2023-11-27T09:25:32.193135+0000","last_active":"2023-11-27T09:25:33.117330+0000","last_peered":"2023-11-27T09:25:33.117330+0000","last_clean":"2023-11-27T09:25:33.117330+0000","last_became_active":"2023-11-27T09:25:32.188083+0000","last_became_peered":"2023-11-27T09:25:32.188083+0000","last_unstale":"2023-11-27T09:25:33.117330+0000","last_undegraded":"2023-11-27T09:25:33.117330+0000","last_fullsized":"2023-11-27T09:25:33.117330+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_clean_scrub_stamp":"2023-11-27T09:25:31.101787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-27T09:25:33.116739+0000","last_change":"2023-11-27T09:25:32.188002+0000","last_active":"2023-11-27T09:25:33.116739+0000","last_peered":"2023-11-27T09:25:33.116739+0000","last_clean":"2023-11-27T09:25:33.116739+0000","last_became_active":"2023-11-27T09:25:32.185114+0000","last_became_peered":"2023-11-27T09:25:32.185114+0000","last_unstale":"2023-11-27T09:25:33.116739+0000","last_undegraded":"2023-11-27T09:25:33.116739+0000","last_fullsized":"2023-11-27T09:25:33.116739+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_clean_scrub_stamp":"2023-11-27T09:25:31.101787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-27T09:25:33.165659+0000","last_change":"2023-11-27T09:25:32.178253+0000","last_active":"2023-11-27T09:25:33.165659+0000","last_peered":"2023-11-27T09:25:33.165659+0000","last_clean":"2023-11-27T09:25:33.165659+0000","last_became_active":"2023-11-27T09:25:32.176381+0000","last_became_peered":"2023-11-27T09:25:32.176381+0000","last_unstale":"2023-11-27T09:25:33.165659+0000","last_undegraded":"2023-11-27T09:25:33.165659+0000","last_fullsized":"2023-11-27T09:25:33.165659+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-27T09:25:31.101787+0000","last_clean_scrub_stamp":"2023-11-27T09:25:31.101787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-27T09:25:33.163465+0000","last_change":"2023-11-27T09:25:31.173601+0000","last_active":"2023-11-27T09:25:33.163465+0000","last_peered":"2023-11-27T09:25:33.163465+0000","last_clean":"2023-11-27T09:25:33.163465+0000","last_became_active":"2023-11-27T09:25:31.170067+0000","last_became_peered":"2023-11-27T09:25:31.170067+0000","last_unstale":"2023-11-27T09:25:33.163465+0000","last_undegraded":"2023-11-27T09:25:33.163465+0000","last_fullsized":"2023-11-27T09:25:33.163465+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-27T09:25:30.098531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-27T09:25:30.098531+0000","last_clean_scrub_stamp":"2023-11-27T09:25:30.098531+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":16996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":29914,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":29914,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-27T09:25:34.320 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-27T09:25:34.320 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-27T09:25:34.320 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-11-27T09:25:34.320 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-27T09:25:34.430 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:34.427+0000 7f4f50500640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:34.432 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:25:34.431+0000 7f4f50500640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:25:34.656 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:34.656 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-27T09:25:34.668 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-11-27T09:25:34.668 INFO:teuthology.run_tasks:Running task cbt... 2023-11-27T09:25:34.680 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi007.front.sepia.ceph.com', 'osds': ['ubuntu@smithi007.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi007.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi007.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi007.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2023-11-27T09:25:34.680 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2023-11-27T09:25:34.697 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:25:34.697 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-11-27T09:25:34.751 DEBUG:teuthology.orchestra.run.smithi007:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2023-11-27T09:25:34.805 INFO:teuthology.orchestra.run.smithi007.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2023-11-27T09:25:35.202 INFO:tasks.cbt:Installing collectl 2023-11-27T09:25:35.202 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2023-11-27T09:25:35.333 INFO:teuthology.orchestra.run.smithi007.stderr:--2023-11-27 09:25:35-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2023-11-27T09:25:35.484 INFO:teuthology.orchestra.run.smithi007.stderr:Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::6812:256f, ... 2023-11-27T09:25:35.494 INFO:teuthology.orchestra.run.smithi007.stderr:Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. 2023-11-27T09:25:35.736 INFO:teuthology.orchestra.run.smithi007.stderr:HTTP request sent, awaiting response... 302 Found 2023-11-27T09:25:35.737 INFO:teuthology.orchestra.run.smithi007.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlZGCPD89MkTPUzaMJ3cYlAV3eQZvCLvHByRgeUZ-8IChpDGIZq7PdcephCCudR3q3ejBl_qhJ0m9WY-egDIaKBNNJcg%3D%3D&use_mirror=phoenixnap&r= [following] 2023-11-27T09:25:35.737 INFO:teuthology.orchestra.run.smithi007.stderr:--2023-11-27 09:25:35-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlZGCPD89MkTPUzaMJ3cYlAV3eQZvCLvHByRgeUZ-8IChpDGIZq7PdcephCCudR3q3ejBl_qhJ0m9WY-egDIaKBNNJcg%3D%3D&use_mirror=phoenixnap&r= 2023-11-27T09:25:35.737 INFO:teuthology.orchestra.run.smithi007.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2023-11-27T09:25:35.808 INFO:teuthology.orchestra.run.smithi007.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2023-11-27T09:25:36.047 INFO:teuthology.orchestra.run.smithi007.stderr:HTTP request sent, awaiting response... 302 Found 2023-11-27T09:25:36.047 INFO:teuthology.orchestra.run.smithi007.stderr:Location: https://phoenixnap.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2023-11-27T09:25:36.047 INFO:teuthology.orchestra.run.smithi007.stderr:--2023-11-27 09:25:36-- https://phoenixnap.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2023-11-27T09:25:36.059 INFO:teuthology.orchestra.run.smithi007.stderr:Resolving phoenixnap.dl.sourceforge.net (phoenixnap.dl.sourceforge.net)... 184.164.141.26 2023-11-27T09:25:36.126 INFO:teuthology.orchestra.run.smithi007.stderr:Connecting to phoenixnap.dl.sourceforge.net (phoenixnap.dl.sourceforge.net)|184.164.141.26|:443... connected. 2023-11-27T09:25:36.437 INFO:teuthology.orchestra.run.smithi007.stderr:HTTP request sent, awaiting response... 200 OK 2023-11-27T09:25:36.437 INFO:teuthology.orchestra.run.smithi007.stderr:Length: 620693 (606K) [application/x-gzip] 2023-11-27T09:25:36.437 INFO:teuthology.orchestra.run.smithi007.stderr:Saving to: ‘collectl.tar.gz’ 2023-11-27T09:25:36.437 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-27T09:25:36.522 INFO:teuthology.orchestra.run.smithi007.stderr: 0K .......... .......... .......... .......... .......... 8% 587K 1s 2023-11-27T09:25:36.570 INFO:teuthology.orchestra.run.smithi007.stderr: 50K .......... .......... .......... .......... .......... 16% 1.01M 1s 2023-11-27T09:25:36.589 INFO:teuthology.orchestra.run.smithi007.stderr: 100K .......... .......... .......... .......... .......... 24% 2.58M 0s 2023-11-27T09:25:36.638 INFO:teuthology.orchestra.run.smithi007.stderr: 150K .......... .......... .......... .......... .......... 32% 1.02M 0s 2023-11-27T09:25:36.658 INFO:teuthology.orchestra.run.smithi007.stderr: 200K .......... .......... .......... .......... .......... 41% 2.63M 0s 2023-11-27T09:25:36.659 INFO:teuthology.orchestra.run.smithi007.stderr: 250K .......... .......... .......... .......... .......... 49% 92.8M 0s 2023-11-27T09:25:36.659 INFO:teuthology.orchestra.run.smithi007.stderr: 300K .......... .......... .......... .......... .......... 57% 113M 0s 2023-11-27T09:25:36.705 INFO:teuthology.orchestra.run.smithi007.stderr: 350K .......... .......... .......... .......... .......... 65% 1.03M 0s 2023-11-27T09:25:36.706 INFO:teuthology.orchestra.run.smithi007.stderr: 400K .......... .......... .......... .......... .......... 74% 144M 0s 2023-11-27T09:25:36.725 INFO:teuthology.orchestra.run.smithi007.stderr: 450K .......... .......... .......... .......... .......... 82% 2.69M 0s 2023-11-27T09:25:36.725 INFO:teuthology.orchestra.run.smithi007.stderr: 500K .......... .......... .......... .......... .......... 90% 96.8M 0s 2023-11-27T09:25:36.725 INFO:teuthology.orchestra.run.smithi007.stderr: 550K .......... .......... .......... .......... .......... 98% 58.2M 0s 2023-11-27T09:25:36.725 INFO:teuthology.orchestra.run.smithi007.stderr: 600K ...... 100% 11.4T=0.3s 2023-11-27T09:25:36.725 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-27T09:25:36.725 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27 09:25:36 (2.06 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2023-11-27T09:25:36.725 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-27T09:25:36.761 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/ 2023-11-27T09:25:36.761 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/collectl.conf 2023-11-27T09:25:36.761 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/ 2023-11-27T09:25:36.761 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/SlowProc.html 2023-11-27T09:25:36.761 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/BuddyInfo.html 2023-11-27T09:25:36.761 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/WhySummary.html 2023-11-27T09:25:36.761 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Socket.html 2023-11-27T09:25:36.761 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/colmux.html 2023-11-27T09:25:36.761 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Tutorial.html 2023-11-27T09:25:36.762 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Gexpr.html 2023-11-27T09:25:36.762 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Import.html 2023-11-27T09:25:36.762 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/style.css 2023-11-27T09:25:36.762 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2023-11-27T09:25:36.762 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2023-11-27T09:25:36.762 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/FileNaming.html 2023-11-27T09:25:36.762 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/TheMath.html 2023-11-27T09:25:36.762 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Environmental.html 2023-11-27T09:25:36.762 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Process.html 2023-11-27T09:25:36.762 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/OutputFiles.html 2023-11-27T09:25:36.762 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/HiResTime.html 2023-11-27T09:25:36.762 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/OutputFormats.html 2023-11-27T09:25:36.762 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/slub.jpg 2023-11-27T09:25:36.763 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Lustre.html 2023-11-27T09:25:36.763 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Documentation.html 2023-11-27T09:25:36.763 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Inodes.html 2023-11-27T09:25:36.763 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Graphite.html 2023-11-27T09:25:36.763 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/InputFiles.html 2023-11-27T09:25:36.763 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Export.html 2023-11-27T09:25:36.763 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Interrupts.html 2023-11-27T09:25:36.763 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Startup.html 2023-11-27T09:25:36.763 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Performance.html 2023-11-27T09:25:36.763 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Messages.html 2023-11-27T09:25:36.763 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Logging.html 2023-11-27T09:25:36.763 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Data-detail.html 2023-11-27T09:25:36.763 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/CPUs.html 2023-11-27T09:25:36.764 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Matrix.html 2023-11-27T09:25:36.764 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Data-brief.html 2023-11-27T09:25:36.764 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/SlabInfo.html 2023-11-27T09:25:36.764 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Misc.html 2023-11-27T09:25:36.764 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Examples.html 2023-11-27T09:25:36.764 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/OpenStack.html 2023-11-27T09:25:36.764 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Plotfiles.html 2023-11-27T09:25:36.764 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2023-11-27T09:25:36.764 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2023-11-27T09:25:36.764 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Data-verbose.html 2023-11-27T09:25:36.764 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Data.html 2023-11-27T09:25:36.764 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Exceptions.html 2023-11-27T09:25:36.764 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/RunningAsAService.html 2023-11-27T09:25:36.764 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2023-11-27T09:25:36.765 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Ganglia.jpg 2023-11-27T09:25:36.765 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/OperationalModes.html 2023-11-27T09:25:36.765 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Playback.html 2023-11-27T09:25:36.765 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Hello.html 2023-11-27T09:25:36.765 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Sockets.html 2023-11-27T09:25:36.765 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Network.html 2023-11-27T09:25:36.765 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Infiniband.html 2023-11-27T09:25:36.765 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Memory.html 2023-11-27T09:25:36.765 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2023-11-27T09:25:36.765 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/Disks.html 2023-11-27T09:25:36.765 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/docs/NfsInfo.html 2023-11-27T09:25:36.765 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/service/ 2023-11-27T09:25:36.765 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/service/collectl.service 2023-11-27T09:25:36.766 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/initd/ 2023-11-27T09:25:36.766 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/initd/collectl-suse 2023-11-27T09:25:36.766 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/initd/collectl-debian 2023-11-27T09:25:36.766 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/initd/collectl-generic 2023-11-27T09:25:36.766 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/initd/collectl 2023-11-27T09:25:36.766 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/misc.ph 2023-11-27T09:25:36.766 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/gexpr.ph 2023-11-27T09:25:36.766 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/statsd.ph 2023-11-27T09:25:36.766 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/envrules.std 2023-11-27T09:25:36.766 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/hello.ph 2023-11-27T09:25:36.766 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/vmsum.ph 2023-11-27T09:25:36.766 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/proctree.ph 2023-11-27T09:25:36.766 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/README-WINDOWS 2023-11-27T09:25:36.767 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/formatit.ph 2023-11-27T09:25:36.768 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/colmux 2023-11-27T09:25:36.768 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/INSTALL 2023-11-27T09:25:36.768 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/graphite.ph 2023-11-27T09:25:36.769 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/RELEASE-collectl 2023-11-27T09:25:36.769 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/vmstat.ph 2023-11-27T09:25:36.769 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/README 2023-11-27T09:25:36.769 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/man1/ 2023-11-27T09:25:36.769 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/man1/collectl.1 2023-11-27T09:25:36.769 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/man1/colmux.1 2023-11-27T09:25:36.769 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/lexpr.ph 2023-11-27T09:25:36.769 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/client.pl 2023-11-27T09:25:36.769 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/collectl 2023-11-27T09:25:36.769 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/COPYING 2023-11-27T09:25:36.769 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/vnet.ph 2023-11-27T09:25:36.769 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/GPL 2023-11-27T09:25:36.769 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/UNINSTALL 2023-11-27T09:25:36.770 INFO:teuthology.orchestra.run.smithi007.stdout:collectl-4.3.1/ARTISTIC 2023-11-27T09:25:36.827 INFO:teuthology.orchestra.run.smithi007.stderr:./INSTALL: line 90: chkconfig: command not found 2023-11-27T09:25:36.829 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2023-11-27T09:25:37.447 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:07:22 ago on Mon 27 Nov 2023 09:18:15 AM UTC. 2023-11-27T09:25:37.547 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2023-11-27T09:25:37.550 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2023-11-27T09:25:37.607 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2023-11-27T09:25:37.608 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:25:37.608 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2023-11-27T09:25:37.608 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:25:37.608 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2023-11-27T09:25:37.608 INFO:teuthology.orchestra.run.smithi007.stdout: librbd-devel x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 23 k 2023-11-27T09:25:37.608 INFO:teuthology.orchestra.run.smithi007.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2023-11-27T09:25:37.608 INFO:teuthology.orchestra.run.smithi007.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2023-11-27T09:25:37.609 INFO:teuthology.orchestra.run.smithi007.stdout: perf x86_64 5.14.0-386.el9 appstream 7.4 M 2023-11-27T09:25:37.609 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2023-11-27T09:25:37.609 INFO:teuthology.orchestra.run.smithi007.stdout: libradospp-devel x86_64 2:18.0.0-7476.g705aae7f.el9 ceph 30 k 2023-11-27T09:25:37.609 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:37.609 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2023-11-27T09:25:37.609 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2023-11-27T09:25:37.609 INFO:teuthology.orchestra.run.smithi007.stdout:Install 5 Packages 2023-11-27T09:25:37.609 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:37.610 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 7.6 M 2023-11-27T09:25:37.610 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 11 M 2023-11-27T09:25:37.610 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2023-11-27T09:25:38.464 INFO:teuthology.orchestra.run.smithi007.stdout:(1/5): librbd-devel-18.0.0-7476.g705aae7f.el9.x 55 kB/s | 23 kB 00:00 2023-11-27T09:25:38.556 INFO:teuthology.orchestra.run.smithi007.stdout:(2/5): libradospp-devel-18.0.0-7476.g705aae7f.e 59 kB/s | 30 kB 00:00 2023-11-27T09:25:38.740 INFO:teuthology.orchestra.run.smithi007.stdout:(3/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 72 kB/s | 13 kB 00:00 2023-11-27T09:25:38.765 INFO:teuthology.orchestra.run.smithi007.stdout:(4/5): pdsh-2.34-7.el9.x86_64.rpm 395 kB/s | 118 kB 00:00 2023-11-27T09:25:38.982 INFO:teuthology.orchestra.run.smithi007.stdout:(5/5): perf-5.14.0-386.el9.x86_64.rpm 7.9 MB/s | 7.4 MB 00:00 2023-11-27T09:25:38.985 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2023-11-27T09:25:38.985 INFO:teuthology.orchestra.run.smithi007.stdout:Total 5.5 MB/s | 7.6 MB 00:01 2023-11-27T09:25:39.065 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2023-11-27T09:25:39.089 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2023-11-27T09:25:39.089 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2023-11-27T09:25:39.302 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2023-11-27T09:25:39.302 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2023-11-27T09:25:39.768 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2023-11-27T09:25:39.835 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2023-11-27T09:25:39.864 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2023-11-27T09:25:39.896 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libradospp-devel-2:18.0.0-7476.g705aae7f.el9.x86_64 3/5 2023-11-27T09:25:40.265 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librbd-devel-2:18.0.0-7476.g705aae7f.el9.x86_64 4/5 2023-11-27T09:25:40.453 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : perf-5.14.0-386.el9.x86_64 5/5 2023-11-27T09:25:41.375 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: perf-5.14.0-386.el9.x86_64 5/5 2023-11-27T09:25:41.375 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libradospp-devel-2:18.0.0-7476.g705aae7f.el9.x86_64 1/5 2023-11-27T09:25:41.375 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd-devel-2:18.0.0-7476.g705aae7f.el9.x86_64 2/5 2023-11-27T09:25:41.375 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : perf-5.14.0-386.el9.x86_64 3/5 2023-11-27T09:25:41.375 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2023-11-27T09:25:41.770 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2023-11-27T09:25:41.771 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:41.771 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2023-11-27T09:25:41.771 INFO:teuthology.orchestra.run.smithi007.stdout: libradospp-devel-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:25:41.771 INFO:teuthology.orchestra.run.smithi007.stdout: librbd-devel-2:18.0.0-7476.g705aae7f.el9.x86_64 2023-11-27T09:25:41.771 INFO:teuthology.orchestra.run.smithi007.stdout: pdsh-2.34-7.el9.x86_64 2023-11-27T09:25:41.771 INFO:teuthology.orchestra.run.smithi007.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2023-11-27T09:25:41.771 INFO:teuthology.orchestra.run.smithi007.stdout: perf-5.14.0-386.el9.x86_64 2023-11-27T09:25:41.771 INFO:teuthology.orchestra.run.smithi007.stdout: 2023-11-27T09:25:41.771 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2023-11-27T09:25:42.179 INFO:tasks.cbt:benchmark: radosbench 2023-11-27T09:25:42.179 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2023-11-27T09:25:42.213 DEBUG:teuthology.orchestra.run.smithi007:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-11-27T09:25:42.599 INFO:teuthology.orchestra.run.smithi007.stderr:09:25:42 - DEBUG - cbt - Settings.cluster: 2023-11-27T09:25:42.599 INFO:teuthology.orchestra.run.smithi007.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2023-11-27T09:25:42.599 INFO:teuthology.orchestra.run.smithi007.stderr: 'clients': ['ubuntu@smithi007.front.sepia.ceph.com'], 2023-11-27T09:25:42.599 INFO:teuthology.orchestra.run.smithi007.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2023-11-27T09:25:42.599 INFO:teuthology.orchestra.run.smithi007.stderr: 'head': 'ubuntu@smithi007.front.sepia.ceph.com', 2023-11-27T09:25:42.599 INFO:teuthology.orchestra.run.smithi007.stderr: 'is_teuthology': True, 2023-11-27T09:25:42.600 INFO:teuthology.orchestra.run.smithi007.stderr: 'iterations': 1, 2023-11-27T09:25:42.600 INFO:teuthology.orchestra.run.smithi007.stderr: 'mons': ['ubuntu@smithi007.front.sepia.ceph.com'], 2023-11-27T09:25:42.600 INFO:teuthology.orchestra.run.smithi007.stderr: 'osds': ['ubuntu@smithi007.front.sepia.ceph.com'], 2023-11-27T09:25:42.600 INFO:teuthology.orchestra.run.smithi007.stderr: 'osds_per_node': 3, 2023-11-27T09:25:42.600 INFO:teuthology.orchestra.run.smithi007.stderr: 'pid_dir': '/var/run/ceph', 2023-11-27T09:25:42.600 INFO:teuthology.orchestra.run.smithi007.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2023-11-27T09:25:42.600 INFO:teuthology.orchestra.run.smithi007.stderr: 'pgp_size': 256, 2023-11-27T09:25:42.600 INFO:teuthology.orchestra.run.smithi007.stderr: 'replication': 'replicated'}}, 2023-11-27T09:25:42.600 INFO:teuthology.orchestra.run.smithi007.stderr: 'rebuild_every_test': False, 2023-11-27T09:25:42.600 INFO:teuthology.orchestra.run.smithi007.stderr: 'rgws': {'ubuntu@smithi007.front.sepia.ceph.com': None}, 2023-11-27T09:25:42.601 INFO:teuthology.orchestra.run.smithi007.stderr: 'tmp_dir': '/tmp/cbt', 2023-11-27T09:25:42.601 INFO:teuthology.orchestra.run.smithi007.stderr: 'use_existing': True, 2023-11-27T09:25:42.601 INFO:teuthology.orchestra.run.smithi007.stderr: 'user': 'ubuntu'} 2023-11-27T09:25:42.620 INFO:teuthology.orchestra.run.smithi007.stderr:09:25:42 - INFO - cbt - Pausing for 60s for idle monitoring. 2023-11-27T09:25:42.622 INFO:teuthology.orchestra.run.smithi007.stderr:09:25:42 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:25:42.622 INFO:teuthology.orchestra.run.smithi007.stderr:09:25:42 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:25:42.622 INFO:teuthology.orchestra.run.smithi007.stderr:09:25:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl 2023-11-27T09:25:42.626 INFO:teuthology.orchestra.run.smithi007.stderr:09:25:42 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:25:42.626 INFO:teuthology.orchestra.run.smithi007.stderr:09:25:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-11-27T09:25:42.627 INFO:teuthology.orchestra.run.smithi007.stderr:09:25:42 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:25:42.627 INFO:teuthology.orchestra.run.smithi007.stderr:09:25:42 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:25:42.627 INFO:teuthology.orchestra.run.smithi007.stderr:09:25:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf 2023-11-27T09:25:42.630 INFO:teuthology.orchestra.run.smithi007.stderr:09:25:42 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:25:42.630 INFO:teuthology.orchestra.run.smithi007.stderr:09:25:42 - DEBUG - cbt - PerfMonitoring: in local_node 2023-11-27T09:25:42.630 INFO:teuthology.orchestra.run.smithi007.stderr:09:25:42 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-11-27T09:26:42.678 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:42.678 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-11-27T09:26:42.710 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:42.710 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-11-27T09:26:42.755 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:42.755 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2023-11-27T09:26:42.757 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:42.757 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2023-11-27T09:26:42.758 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:26:42.759 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:42.759 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/* 2023-11-27T09:26:42.780 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:42.780 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/$(basename ${i}).smithi007.front.sepia.ceph.com ; done 2023-11-27T09:26:42.785 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:26:42.785 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:42.786 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo killall -9 rados 2023-11-27T09:26:42.818 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - WARNING - cbt - sudo killall -9 rados 2023-11-27T09:26:42.818 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - WARNING - cbt - error 1 seen, continuing anyway... 2023-11-27T09:26:42.838 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3. 2023-11-27T09:26:42.838 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004 2023-11-27T09:26:42.839 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:26:42.839 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:42.839 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004 2023-11-27T09:26:42.858 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:26:42.858 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:42.858 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl 2023-11-27T09:26:42.861 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:42.861 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-11-27T09:26:42.863 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:26:42.863 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:42.863 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf 2023-11-27T09:26:42.866 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:42.866 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - PerfMonitoring: in local_node 2023-11-27T09:26:42.866 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-11-27T09:26:42.866 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:26:42.866 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:42.866 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:42 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2023-11-27T09:26:43.172 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:43 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:26:43.172 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:43 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:43.172 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:43 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2023-11-27T09:26:43.597 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:43 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:26:43.597 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:43 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:43.597 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:43 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2023-11-27T09:26:44.599 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:44 - INFO - cbt - Checking Health after pool creation. 2023-11-27T09:26:49.604 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:49 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:26:49.604 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:49 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:49.604 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:49 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-11-27T09:26:49.955 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:49 - INFO - cbt - Final Pool Health Check. 2023-11-27T09:26:54.960 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:54 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:26:54.961 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:54 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:54.961 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-11-27T09:26:55.285 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:55 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:55.285 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-11-27T09:26:55.303 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:55 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:55.303 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-11-27T09:26:55.339 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:55 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:55.339 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2023-11-27T09:26:55.341 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:55 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:55.341 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2023-11-27T09:26:55.343 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:55 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:26:55.343 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:55 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:55.343 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2023-11-27T09:26:55.606 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:55 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:55.606 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2023-11-27T09:26:56.190 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:56 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:26:56.190 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:56 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:56.190 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2023-11-27T09:26:56.532 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:56 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:26:56.532 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:56 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:56.532 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:56 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write 2023-11-27T09:26:56.600 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:56 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:26:56.600 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:56 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:56.601 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/ceph_settings.out 2023-11-27T09:26:57.412 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:57 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:26:57.412 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:57 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:57.413 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/collectl 2023-11-27T09:26:57.415 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:57 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:57.415 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-11-27T09:26:57.416 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:57 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:26:57.417 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:57 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:57.417 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf 2023-11-27T09:26:57.420 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:57 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:57.420 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:57 - DEBUG - cbt - PerfMonitoring: in local_node 2023-11-27T09:26:57.420 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:57 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-11-27T09:26:57.420 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:57 - INFO - cbt - Running radosbench write test. 2023-11-27T09:26:57.420 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:57 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:26:57.420 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:57 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:57.420 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/output.0 2023-11-27T09:26:57.422 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:57 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:26:57.422 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:57 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:26:57.422 INFO:teuthology.orchestra.run.smithi007.stderr:09:26:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/output.1 2023-11-27T09:27:58.537 INFO:teuthology.orchestra.run.smithi007.stderr:09:27:58 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:27:59.129 INFO:teuthology.orchestra.run.smithi007.stderr:09:27:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-11-27T09:27:59.131 INFO:teuthology.orchestra.run.smithi007.stderr:09:27:58 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:27:59.131 INFO:teuthology.orchestra.run.smithi007.stderr:09:27:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-11-27T09:27:59.131 INFO:teuthology.orchestra.run.smithi007.stderr:09:27:58 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:27:59.131 INFO:teuthology.orchestra.run.smithi007.stderr:09:27:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf/perf.data 2023-11-27T09:27:59.131 INFO:teuthology.orchestra.run.smithi007.stderr:09:27:58 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:27:59.131 INFO:teuthology.orchestra.run.smithi007.stderr:09:27:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf/perf_stat.* 2023-11-27T09:27:59.131 INFO:teuthology.orchestra.run.smithi007.stderr:09:27:58 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:27:59.131 INFO:teuthology.orchestra.run.smithi007.stderr:09:27:58 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:27:59.131 INFO:teuthology.orchestra.run.smithi007.stderr:09:27:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/historic_ops.out 2023-11-27T09:27:59.131 INFO:teuthology.orchestra.run.smithi007.stderr:09:27:58 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:27:59.132 INFO:teuthology.orchestra.run.smithi007.stderr:09:27:58 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:27:59.132 INFO:teuthology.orchestra.run.smithi007.stderr:09:27:58 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/* 2023-11-27T09:27:59.132 INFO:teuthology.orchestra.run.smithi007.stderr:09:27:58 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:27:59.132 INFO:teuthology.orchestra.run.smithi007.stderr:09:27:58 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/$(basename ${i}).smithi007.front.sepia.ceph.com ; done 2023-11-27T09:27:59.132 INFO:teuthology.orchestra.run.smithi007.stderr:09:27:58 - INFO - cbt - Convert results to json format. 2023-11-27T09:27:59.132 INFO:teuthology.orchestra.run.smithi007.stderr:09:27:58 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:27:59.132 INFO:teuthology.orchestra.run.smithi007.stderr:09:27:58 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:27:59.132 INFO:teuthology.orchestra.run.smithi007.stderr:09:27:58 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:27:59.132 INFO:teuthology.orchestra.run.smithi007.stderr:09:27:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2023-11-27T09:27:59.786 INFO:teuthology.orchestra.run.smithi007.stderr:09:27:59 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:27:59.786 INFO:teuthology.orchestra.run.smithi007.stderr:09:27:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2023-11-27T09:28:00.012 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:00 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:28:00.013 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:00 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:28:00.013 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2023-11-27T09:28:00.348 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:00 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:28:00.349 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:00 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:28:00.349 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:00 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand 2023-11-27T09:28:00.380 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:00 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:28:00.380 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:00 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:28:00.380 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/ceph_settings.out 2023-11-27T09:28:01.110 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:01 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:28:01.110 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:01 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:28:01.110 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/collectl 2023-11-27T09:28:01.113 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:01 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:28:01.113 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-11-27T09:28:01.114 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:01 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:28:01.115 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:01 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:28:01.115 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/perf 2023-11-27T09:28:01.118 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:01 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:28:01.118 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:01 - DEBUG - cbt - PerfMonitoring: in local_node 2023-11-27T09:28:01.118 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:01 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-11-27T09:28:01.118 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:01 - INFO - cbt - Running radosbench rand test. 2023-11-27T09:28:01.118 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:01 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:28:01.118 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:01 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:28:01.118 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 rand --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/output.0 2023-11-27T09:28:01.120 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:01 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:28:01.120 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:01 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:28:01.120 INFO:teuthology.orchestra.run.smithi007.stderr:09:28:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 rand --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/output.1 2023-11-27T09:29:02.245 INFO:teuthology.orchestra.run.smithi007.stderr:09:29:02 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:29:02.245 INFO:teuthology.orchestra.run.smithi007.stderr:09:29:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-11-27T09:29:02.270 INFO:teuthology.orchestra.run.smithi007.stderr:09:29:02 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:29:02.270 INFO:teuthology.orchestra.run.smithi007.stderr:09:29:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-11-27T09:29:02.305 INFO:teuthology.orchestra.run.smithi007.stderr:09:29:02 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:29:02.305 INFO:teuthology.orchestra.run.smithi007.stderr:09:29:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/perf/perf.data 2023-11-27T09:29:02.307 INFO:teuthology.orchestra.run.smithi007.stderr:09:29:02 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:29:02.307 INFO:teuthology.orchestra.run.smithi007.stderr:09:29:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/perf/perf_stat.* 2023-11-27T09:29:02.308 INFO:teuthology.orchestra.run.smithi007.stderr:09:29:02 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:29:02.309 INFO:teuthology.orchestra.run.smithi007.stderr:09:29:02 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:29:02.309 INFO:teuthology.orchestra.run.smithi007.stderr:09:29:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/historic_ops.out 2023-11-27T09:29:02.642 INFO:teuthology.orchestra.run.smithi007.stderr:09:29:02 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:29:02.643 INFO:teuthology.orchestra.run.smithi007.stderr:09:29:02 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:29:02.643 INFO:teuthology.orchestra.run.smithi007.stderr:09:29:02 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/* 2023-11-27T09:29:02.662 INFO:teuthology.orchestra.run.smithi007.stderr:09:29:02 - DEBUG - cbt - get_fqdn_local()=smithi007.front.sepia.ceph.com 2023-11-27T09:29:02.662 INFO:teuthology.orchestra.run.smithi007.stderr:09:29:02 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/$(basename ${i}).smithi007.front.sepia.ceph.com ; done 2023-11-27T09:29:02.694 INFO:teuthology.orchestra.run.smithi007.stderr:09:29:02 - INFO - cbt - Convert results to json format. 2023-11-27T09:29:02.694 INFO:teuthology.orchestra.run.smithi007.stderr:09:29:02 - DEBUG - cbt - Nodes : ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:29:02.725 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2023-11-27T09:29:02.734 DEBUG:teuthology.orchestra.run.smithi007:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2023-11-27T09:29:02.918 INFO:tasks.cbt_performance:Collecting CBT performance data config 2023-11-27T09:29:02.918 INFO:tasks.cbt_performance:reading cbt preferences 2023-11-27T09:29:02.919 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -name 'json_output.*' 2023-11-27T09:29:02.982 INFO:teuthology.orchestra.run.smithi007.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/json_output.1.smithi007.front.sepia.ceph.com 2023-11-27T09:29:02.982 INFO:teuthology.orchestra.run.smithi007.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/json_output.0.smithi007.front.sepia.ceph.com 2023-11-27T09:29:02.982 INFO:teuthology.orchestra.run.smithi007.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/json_output.1.smithi007.front.sepia.ceph.com 2023-11-27T09:29:02.982 INFO:teuthology.orchestra.run.smithi007.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/json_output.0.smithi007.front.sepia.ceph.com 2023-11-27T09:29:02.983 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:29:02.983 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/json_output.1.smithi007.front.sepia.ceph.com of=/dev/stdout 2023-11-27T09:29:03.017 INFO:tasks.cbt_performance:reading total cpu cycles 2023-11-27T09:29:03.017 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write -name 'perf_stat.*' 2023-11-27T09:29:03.072 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-11-27T09:29:03.073 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-11-27T09:29:03.073 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:29:03.073 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/json_output.0.smithi007.front.sepia.ceph.com of=/dev/stdout 2023-11-27T09:29:03.128 INFO:tasks.cbt_performance:reading total cpu cycles 2023-11-27T09:29:03.128 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write -name 'perf_stat.*' 2023-11-27T09:29:03.182 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-11-27T09:29:03.182 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-11-27T09:29:03.183 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:29:03.183 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/json_output.1.smithi007.front.sepia.ceph.com of=/dev/stdout 2023-11-27T09:29:03.237 INFO:tasks.cbt_performance:reading total cpu cycles 2023-11-27T09:29:03.237 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand -name 'perf_stat.*' 2023-11-27T09:29:03.293 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-11-27T09:29:03.293 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-11-27T09:29:03.293 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2023-11-27T09:29:03.293 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/json_output.0.smithi007.front.sepia.ceph.com of=/dev/stdout 2023-11-27T09:29:03.347 INFO:tasks.cbt_performance:reading total cpu cycles 2023-11-27T09:29:03.347 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand -name 'perf_stat.*' 2023-11-27T09:29:03.402 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-11-27T09:29:03.402 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-11-27T09:29:03.412 INFO:tasks.cbt_performance:Data inserted successfully. 2023-11-27T09:29:03.418 INFO:tasks.cbt_performance:Data inserted successfully. 2023-11-27T09:29:03.424 INFO:tasks.cbt_performance:Data inserted successfully. 2023-11-27T09:29:03.430 INFO:tasks.cbt_performance:Data inserted successfully. 2023-11-27T09:29:03.430 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-27T09:29:03.439 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-11-27T09:29:03.554 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:29:03.547+0000 7fcd90c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:29:03.556 INFO:teuthology.orchestra.run.smithi007.stderr:2023-11-27T09:29:03.555+0000 7fcd90c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-27T09:29:03.755 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-27T09:29:03.755 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-27T09:29:03.755 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-11-27T09:29:03.756 INFO:teuthology.orchestra.run:waiting for 300 2023-11-27T09:29:07.140 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-11-27T09:29:09.856 INFO:tasks.ceph.osd.0:Stopped 2023-11-27T09:29:09.857 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-11-27T09:29:09.857 INFO:teuthology.orchestra.run:waiting for 300 2023-11-27T09:29:12.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-11-27T09:29:12.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-11-27T09:29:15.958 INFO:tasks.ceph.osd.1:Stopped 2023-11-27T09:29:15.958 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-11-27T09:29:15.959 INFO:teuthology.orchestra.run:waiting for 300 2023-11-27T09:29:17.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-11-27T09:29:17.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-11-27T09:29:22.060 INFO:tasks.ceph.osd.2:Stopped 2023-11-27T09:29:22.060 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-27T09:29:22.061 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-11-27T09:29:22.061 INFO:teuthology.orchestra.run:waiting for 300 2023-11-27T09:29:22.092 INFO:tasks.ceph.mgr.x:Stopped 2023-11-27T09:29:22.092 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-27T09:29:22.093 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-11-27T09:29:22.093 INFO:teuthology.orchestra.run:waiting for 300 2023-11-27T09:29:22.146 INFO:tasks.ceph.mon.a:Stopped 2023-11-27T09:29:22.146 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-27T09:29:22.146 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-11-27T09:29:22.243 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:29:22.243 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-11-27T09:29:22.506 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:29:22.506 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-11-27T09:29:22.584 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:29:22.584 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-11-27T09:29:22.641 INFO:tasks.ceph:Archiving mon data... 2023-11-27T09:29:22.642 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467536/data/mon.a.tgz 2023-11-27T09:29:22.642 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2023-11-27T09:29:22.645 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-11-27T09:29:22.658 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.LvoSPnLcsR 2023-11-27T09:29:22.658 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.LvoSPnLcsR 2023-11-27T09:29:22.781 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.LvoSPnLcsR 2023-11-27T09:29:22.876 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.LvoSPnLcsR is 186KB 2023-11-27T09:29:22.900 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.LvoSPnLcsR 2023-11-27T09:29:22.914 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-27T09:29:22.914 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-27T09:29:22.984 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-11-27T09:29:22.984 DEBUG:teuthology.orchestra.run.smithi007:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-27T09:29:23.057 INFO:teuthology.orchestra.run.smithi007.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-27T09:29:23.062 INFO:tasks.ceph:Archiving crash dumps... 2023-11-27T09:29:23.063 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467536/remote/smithi007/crash 2023-11-27T09:29:23.064 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-27T09:29:23.131 INFO:tasks.ceph:Compressing logs... 2023-11-27T09:29:23.131 DEBUG:teuthology.orchestra.run.smithi007:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-27T09:29:23.219 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82775.log 2023-11-27T09:29:23.219 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-11-27T09:29:23.219 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.82775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83195.log 2023-11-27T09:29:23.219 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82940.log 2023-11-27T09:29:23.221 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80372.log 2023-11-27T09:29:23.221 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81981.log 2023-11-27T09:29:23.221 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82941.log 2023-11-27T09:29:23.221 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82672.log 2023-11-27T09:29:23.221 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84292.log 2023-11-27T09:29:23.221 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-27T09:29:23.222 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84014.log 2023-11-27T09:29:23.222 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82817.log 2023-11-27T09:29:23.222 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82859.log 2023-11-27T09:29:23.225 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84291.log 2023-11-27T09:29:23.225 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83065.log 2023-11-27T09:29:23.225 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84060.log 2023-11-27T09:29:23.225 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-11-27T09:29:23.225 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82021.log 2023-11-27T09:29:23.225 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82179.log 2023-11-27T09:29:23.226 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83237.log 2023-11-27T09:29:23.226 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84592.log 2023-11-27T09:29:23.226 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83923.log 2023-11-27T09:29:23.226 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82714.log 2023-11-27T09:29:23.226 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81941.log 2023-11-27T09:29:23.226 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84037.log 2023-11-27T09:29:23.226 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83153.log 2023-11-27T09:29:23.226 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83985.log 2023-11-27T09:29:23.226 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-11-27T09:29:23.226 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82616.log 2023-11-27T09:29:23.228 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-11-27T09:29:23.229 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84126.log 2023-11-27T09:29:23.229 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83066.log 2023-11-27T09:29:23.229 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-27T09:29:23.229 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-11-27T09:29:23.229 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81831.log 2023-11-27T09:29:23.229 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82939.log 2023-11-27T09:29:23.229 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82207.log 2023-11-27T09:29:23.229 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84125.log 2023-11-27T09:29:23.229 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81743.log 2023-11-27T09:29:23.229 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83067.log 2023-11-27T09:29:23.238 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph.tmp-client.admin.80372.log: /var/log/ceph/ceph-client.admin.84292.log: /var/log/ceph/ceph-client.admin.81831.log: /var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.82207.log: /var/log/ceph/ceph-client.admin.84291.log: /var/log/ceph/ceph-client.admin.84125.log: /var/log/ceph/ceph-client.admin.81743.log: /var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.84592.log: /var/log/ceph/ceph-mgr.x.log: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.84292.log.gz 2023-11-27T09:29:23.238 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph.log: 65.6% -- replaced with /var/log/ceph/ceph-client.admin.84291.log.gz 2023-11-27T09:29:23.238 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.2.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82775.log.gz 2023-11-27T09:29:23.238 INFO:teuthology.orchestra.run.smithi007.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.84592.log.gz 2023-11-27T09:29:23.239 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.82940.log: /var/log/ceph/ceph-client.admin.83195.log: /var/log/ceph/ceph-client.admin.82859.log: /var/log/ceph/ceph-client.admin.83065.log: /var/log/ceph/ceph-client.admin.82672.log: /var/log/ceph/ceph-client.admin.82817.log: /var/log/ceph/ceph-client.admin.81981.log: /var/log/ceph/ceph-client.admin.82941.log: /var/log/ceph/ceph-client.admin.84014.log: /var/log/ceph/ceph-client.admin.82714.log: /var/log/ceph/ceph-client.admin.83985.log: /var/log/ceph/ceph-client.admin.82021.log: /var/log/ceph/ceph-client.admin.84060.log: /var/log/ceph/ceph-client.admin.82179.log: /var/log/ceph/ceph-client.admin.83237.log: /var/log/ceph/ceph-client.admin.82616.log: /var/log/ceph/ceph-client.admin.84037.log: /var/log/ceph/ceph-client.admin.83923.log: /var/log/ceph/ceph-client.admin.81941.log: /var/log/ceph/ceph-client.admin.83066.log: /var/log/ceph/ceph-client.admin.83153.log: /var/log/ceph/ceph-client.admin.84126.log: 86.0% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-27T09:29:23.241 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.82939.log: /var/log/ceph/ceph-client.admin.83067.log: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-11-27T09:29:23.249 INFO:teuthology.orchestra.run.smithi007.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.84125.log.gz 2023-11-27T09:29:23.249 INFO:teuthology.orchestra.run.smithi007.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.84126.log.gz 2023-11-27T09:29:23.250 INFO:teuthology.orchestra.run.smithi007.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81743.log.gz 2023-11-27T09:29:23.250 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81831.log.gz 2023-11-27T09:29:23.252 INFO:teuthology.orchestra.run.smithi007.stderr: 87.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-27T09:29:23.252 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81941.log.gz 2023-11-27T09:29:23.252 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81981.log.gz 2023-11-27T09:29:23.252 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82021.log.gz 2023-11-27T09:29:23.252 INFO:teuthology.orchestra.run.smithi007.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82179.log.gz 2023-11-27T09:29:23.252 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82207.log.gz 2023-11-27T09:29:23.253 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82616.log.gz 2023-11-27T09:29:23.253 INFO:teuthology.orchestra.run.smithi007.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82714.log.gz 49.6% 2023-11-27T09:29:23.253 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.82672.log.gz 2023-11-27T09:29:23.253 INFO:teuthology.orchestra.run.smithi007.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.84014.log.gz 2023-11-27T09:29:23.253 INFO:teuthology.orchestra.run.smithi007.stderr: 50.4% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83923.log.gz -- replaced with /var/log/ceph/ceph-client.admin.83985.log.gz 2023-11-27T09:29:23.253 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-27T09:29:23.253 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.84037.log.gz 2023-11-27T09:29:23.253 INFO:teuthology.orchestra.run.smithi007.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.84060.log.gz 2023-11-27T09:29:23.257 INFO:teuthology.orchestra.run.smithi007.stderr: 50.4% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82859.log.gz -- replaced with /var/log/ceph/ceph-client.admin.82817.log.gz 2023-11-27T09:29:23.257 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-27T09:29:23.266 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82941.log.gz 2023-11-27T09:29:23.266 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82940.log.gz -- replaced with /var/log/ceph/ceph-client.admin.83065.log.gz 2023-11-27T09:29:23.267 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-27T09:29:23.267 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83195.log.gz 50.0% 2023-11-27T09:29:23.267 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.83067.log.gz 2023-11-27T09:29:23.267 INFO:teuthology.orchestra.run.smithi007.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83153.log.gz 50.0% 2023-11-27T09:29:23.267 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.83066.log.gz 2023-11-27T09:29:23.267 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.82939.log.gz 2023-11-27T09:29:23.267 INFO:teuthology.orchestra.run.smithi007.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83237.log.gz 2023-11-27T09:29:23.298 INFO:teuthology.orchestra.run.smithi007.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80372.log.gz 2023-11-27T09:29:23.407 INFO:teuthology.orchestra.run.smithi007.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-11-27T09:29:28.941 INFO:teuthology.orchestra.run.smithi007.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-11-27T09:29:29.157 INFO:teuthology.orchestra.run.smithi007.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-11-27T09:29:29.319 INFO:teuthology.orchestra.run.smithi007.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-11-27T09:29:29.321 INFO:teuthology.orchestra.run.smithi007.stderr: 2023-11-27T09:29:29.321 INFO:teuthology.orchestra.run.smithi007.stderr:real 0m6.135s 2023-11-27T09:29:29.321 INFO:teuthology.orchestra.run.smithi007.stderr:user 0m3.732s 2023-11-27T09:29:29.321 INFO:teuthology.orchestra.run.smithi007.stderr:sys 0m0.190s 2023-11-27T09:29:29.321 INFO:tasks.ceph:Archiving logs... 2023-11-27T09:29:29.322 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/log/ceph to /home/teuthworker/archive/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467536/remote/smithi007/log 2023-11-27T09:29:29.323 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-27T09:29:30.028 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-27T09:29:30.038 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-27T09:29:30.038 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-27T09:29:30.102 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-27T09:29:30.111 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-27T09:29:30.111 DEBUG:teuthology.orchestra.run.smithi007:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-27T09:29:30.132 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2023-11-27T09:29:30.200 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-27T09:29:30.200 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2023-11-27T09:29:30.200 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 45 +1423us[+1424us] +/- 76ms 2023-11-27T09:29:30.201 INFO:teuthology.orchestra.run.smithi007.stdout:^* hv02.front.sepia.ceph.com 3 6 377 42 +31us[ +32us] +/- 63ms 2023-11-27T09:29:30.201 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 -1307us[-1305us] +/- 64ms 2023-11-27T09:29:30.201 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-27T09:29:30.212 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-27T09:29:30.220 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-27T09:29:30.221 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-27T09:29:30.228 DEBUG:teuthology.orchestra.run.smithi007:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-27T09:29:30.420 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-27T09:29:30.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-27T09:29:30.456 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-27T09:29:30.465 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-27T09:29:30.474 INFO:teuthology.task.internal:Duration was 1065.631668 seconds 2023-11-27T09:29:30.475 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-27T09:29:30.484 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-27T09:29:30.484 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-27T09:29:30.634 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-27T09:29:31.055 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-27T09:29:31.055 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi007.front.sepia.ceph.com 2023-11-27T09:29:31.055 DEBUG:teuthology.orchestra.run.smithi007:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-27T09:29:31.076 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-27T09:29:31.076 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-27T09:29:31.147 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-27T09:29:31.148 DEBUG:teuthology.orchestra.run.smithi007:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-27T09:29:31.287 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-27T09:29:31.297 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-27T09:29:31.297 DEBUG:teuthology.orchestra.run.smithi007:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-27T09:29:31.334 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-27T09:29:31.344 DEBUG:teuthology.orchestra.run.smithi007:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-27T09:29:31.415 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = core 2023-11-27T09:29:31.442 DEBUG:teuthology.orchestra.run.smithi007:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-27T09:29:31.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-27T09:29:31.497 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-27T09:29:31.507 INFO:teuthology.task.internal:Transferring archived files... 2023-11-27T09:29:31.507 DEBUG:teuthology.misc:Transferring archived files from smithi007:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467536/remote/smithi007 2023-11-27T09:29:31.508 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-27T09:29:31.635 INFO:teuthology.task.internal:Removing archive directory... 2023-11-27T09:29:31.635 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-27T09:29:31.688 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-27T09:29:31.701 INFO:teuthology.task.internal:Not uploading archives. 2023-11-27T09:29:31.702 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-27T09:29:31.711 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-27T09:29:31.712 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-27T09:29:31.743 INFO:teuthology.orchestra.run.smithi007.stdout: 265884 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 27 09:29 /home/ubuntu/cephtest 2023-11-27T09:29:31.744 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-27T09:29:31.755 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-27T09:29:31.778 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/Rfy7LkB4k/rados-bench?orgId=1&var-branch_name=wip-yingxin-crimson-make-crosscore-send-ordered&var-job_id_selected=7467536 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_rand_read} duration: 1065.631668329239 flavor: crimson owner: scheduled_yingxin@teuthology success: true 2023-11-27T09:29:31.778 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-27T09:29:31.850 INFO:teuthology.run:pass