2024-03-29T02:03:21.880 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T02:03:21.881 DEBUG:teuthology.run:Teuthology command: teuthology --name yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi --owner scheduled_yingxin@teuthology --archive /home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629056 --verbose --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_rand_read} -- /home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629056/orig.config.yaml 2024-03-29T02:03:21.927 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T02:03:22.020 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629056 branch: ci-yingxin-crimson-load-obc-3 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_rand_read} email: yingxin.cheng@intel.com first_in_suite: false job_id: '7629056' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: ci-yingxin-crimson-load-obc-3 ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: ci-yingxin-crimson-load-obc-3 env: CRIMSON_COMPAT: '1' sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 owner: scheduled_yingxin@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 sleep_before_teardown: 0 suite: crimson-rados suite_branch: ci-yingxin-crimson-load-obc-3 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 targets: smithi042.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFCz6C5cOWFTqJqpj0ql4JPdLVDLqBcXUMM+G+y8b/mr3qrh/KxRgnL5ln2HBa98TihERnoEY1x4X2AZXE+15ZY= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4096 pool_monitoring_list: - collectl pool_profile: replicated readmode: rand run_monitoring_list: - collectl time: 60 write_only: false cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-29_01:54:41 tube: smithi user: yingxin verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T02:03:22.020 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa; will attempt to use it 2024-03-29T02:03:22.021 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa/tasks 2024-03-29T02:03:22.021 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T02:03:22.022 INFO:teuthology.task.internal:Checking packages... 2024-03-29T02:03:22.046 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '785fc5e8d093374f751c7da3e51e9b56b3718d17' 2024-03-29T02:03:22.046 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T02:03:22.046 INFO:teuthology.packaging:ref: None 2024-03-29T02:03:22.046 INFO:teuthology.packaging:tag: None 2024-03-29T02:03:22.046 INFO:teuthology.packaging:branch: ci-yingxin-crimson-load-obc-3 2024-03-29T02:03:22.046 INFO:teuthology.packaging:sha1: 785fc5e8d093374f751c7da3e51e9b56b3718d17 2024-03-29T02:03:22.047 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=ci-yingxin-crimson-load-obc-3 2024-03-29T02:03:22.189 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2473.g785fc5e8 2024-03-29T02:03:22.190 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T02:03:22.196 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T02:03:22.196 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T02:03:22.216 INFO:teuthology.task.internal:Saving configuration 2024-03-29T02:03:22.235 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T02:03:22.240 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T02:03:22.266 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi042.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629056', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 01:58:29.307183', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFCz6C5cOWFTqJqpj0ql4JPdLVDLqBcXUMM+G+y8b/mr3qrh/KxRgnL5ln2HBa98TihERnoEY1x4X2AZXE+15ZY='} 2024-03-29T02:03:22.266 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T02:03:22.273 INFO:teuthology.task.internal:roles: ubuntu@smithi042.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-03-29T02:03:22.273 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T02:03:22.311 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbaa1646f70>, signals=[15]) 2024-03-29T02:03:22.311 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T02:03:22.317 INFO:teuthology.task.internal:Opening connections... 2024-03-29T02:03:22.317 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:03:22.319 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T02:03:22.426 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T02:03:22.432 DEBUG:teuthology.orchestra.run.smithi042:> uname -m 2024-03-29T02:03:22.494 INFO:teuthology.orchestra.run.smithi042.stdout:x86_64 2024-03-29T02:03:22.495 DEBUG:teuthology.orchestra.run.smithi042:> cat /etc/os-release 2024-03-29T02:03:22.552 INFO:teuthology.orchestra.run.smithi042.stdout:NAME="CentOS Stream" 2024-03-29T02:03:22.552 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION="9" 2024-03-29T02:03:22.552 INFO:teuthology.orchestra.run.smithi042.stdout:ID="centos" 2024-03-29T02:03:22.552 INFO:teuthology.orchestra.run.smithi042.stdout:ID_LIKE="rhel fedora" 2024-03-29T02:03:22.552 INFO:teuthology.orchestra.run.smithi042.stdout:VERSION_ID="9" 2024-03-29T02:03:22.552 INFO:teuthology.orchestra.run.smithi042.stdout:PLATFORM_ID="platform:el9" 2024-03-29T02:03:22.552 INFO:teuthology.orchestra.run.smithi042.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T02:03:22.552 INFO:teuthology.orchestra.run.smithi042.stdout:ANSI_COLOR="0;31" 2024-03-29T02:03:22.552 INFO:teuthology.orchestra.run.smithi042.stdout:LOGO="fedora-logo-icon" 2024-03-29T02:03:22.552 INFO:teuthology.orchestra.run.smithi042.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T02:03:22.552 INFO:teuthology.orchestra.run.smithi042.stdout:HOME_URL="https://centos.org/" 2024-03-29T02:03:22.552 INFO:teuthology.orchestra.run.smithi042.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T02:03:22.553 INFO:teuthology.orchestra.run.smithi042.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T02:03:22.553 INFO:teuthology.orchestra.run.smithi042.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T02:03:22.553 INFO:teuthology.lock.ops:Updating smithi042.front.sepia.ceph.com on lock server 2024-03-29T02:03:22.581 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T02:03:22.589 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T02:03:22.594 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T02:03:22.594 DEBUG:teuthology.orchestra.run.smithi042:> test '!' -e /home/ubuntu/cephtest 2024-03-29T02:03:22.610 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T02:03:22.616 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T02:03:22.616 DEBUG:teuthology.orchestra.run.smithi042:> test -z $(ls -A /var/lib/ceph) 2024-03-29T02:03:22.688 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T02:03:22.720 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T02:03:22.734 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T02:03:22.734 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T02:03:22.735 DEBUG:teuthology.orchestra.run.smithi042:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T02:03:22.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T02:03:22.752 DEBUG:teuthology.orchestra.run.smithi042:> uname -r 2024-03-29T02:03:22.809 INFO:teuthology.orchestra.run.smithi042.stdout:5.14.0-361.el9.x86_64 2024-03-29T02:03:22.810 INFO:teuthology.task.kernel:Running kernel on smithi042: 5.14.0-361.el9.x86_64 2024-03-29T02:03:22.810 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum install -y kernel 2024-03-29T02:03:25.045 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-29T02:03:26.562 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2024-03-29T02:03:29.287 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 9 - BaseOS 3.4 MB/s | 8.0 MB 00:02 2024-03-29T02:03:31.742 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-03-29T02:03:36.264 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 9 - CRB 7.4 MB/s | 6.1 MB 00:00 2024-03-29T02:03:37.689 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 9 - Extras packages 65 kB/s | 16 kB 00:00 2024-03-29T02:03:39.824 INFO:teuthology.orchestra.run.smithi042.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-03-29T02:03:44.967 INFO:teuthology.orchestra.run.smithi042.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-29T02:03:47.323 INFO:teuthology.orchestra.run.smithi042.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T02:03:47.323 INFO:teuthology.orchestra.run.smithi042.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T02:03:47.426 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:03:47.435 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:03:47.435 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repository Size 2024-03-29T02:03:47.435 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:03:47.435 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-29T02:03:47.435 INFO:teuthology.orchestra.run.smithi042.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T02:03:47.435 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-03-29T02:03:47.435 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T02:03:47.435 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T02:03:47.435 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T02:03:47.435 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:03:47.435 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-29T02:03:47.436 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:03:47.436 INFO:teuthology.orchestra.run.smithi042.stdout:Install 4 Packages 2024-03-29T02:03:47.436 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:03:47.436 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 76 M 2024-03-29T02:03:47.436 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 123 M 2024-03-29T02:03:47.437 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-29T02:03:47.593 INFO:teuthology.orchestra.run.smithi042.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T02:03:48.443 INFO:teuthology.orchestra.run.smithi042.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 15 MB/s | 15 MB 00:01 2024-03-29T02:03:49.076 INFO:teuthology.orchestra.run.smithi042.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 20 MB/s | 34 MB 00:01 2024-03-29T02:03:49.885 INFO:teuthology.orchestra.run.smithi042.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 12 MB/s | 28 MB 00:02 2024-03-29T02:03:49.888 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:03:49.888 INFO:teuthology.orchestra.run.smithi042.stdout:Total 31 MB/s | 76 MB 00:02 2024-03-29T02:03:49.888 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-29T02:03:50.338 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-29T02:03:50.338 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-29T02:03:51.208 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-29T02:03:51.208 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-29T02:03:53.632 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-29T02:03:54.291 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T02:03:54.379 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T02:03:57.295 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T02:03:57.434 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T02:04:02.508 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T02:04:02.550 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T02:04:07.648 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T02:04:38.450 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T02:04:38.451 INFO:teuthology.orchestra.run.smithi042.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T02:04:38.451 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:04:38.495 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T02:04:38.941 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T02:04:38.942 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T02:04:38.942 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T02:04:38.942 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T02:04:39.411 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T02:04:39.411 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:04:39.411 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-29T02:04:39.411 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T02:04:39.411 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T02:04:39.411 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T02:04:39.411 INFO:teuthology.orchestra.run.smithi042.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T02:04:39.411 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:04:39.411 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:04:39.756 DEBUG:teuthology.orchestra.run.smithi042:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T02:04:39.870 INFO:teuthology.orchestra.run.smithi042.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T02:04:39.870 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T02:04:39.870 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T02:04:39.871 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T02:04:39.871 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T02:04:39.871 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T02:04:39.871 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi042.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T02:04:39.871 DEBUG:teuthology.orchestra.run.smithi042:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T02:04:39.954 INFO:teuthology.orchestra.run.smithi042.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T02:04:39.954 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T02:04:39.954 DEBUG:teuthology.orchestra.run.smithi042:> sudo rpm -qi grub2-tools 2024-03-29T02:04:40.002 INFO:teuthology.orchestra.run.smithi042.stdout:Name : grub2-tools 2024-03-29T02:04:40.002 INFO:teuthology.orchestra.run.smithi042.stdout:Epoch : 1 2024-03-29T02:04:40.002 INFO:teuthology.orchestra.run.smithi042.stdout:Version : 2.06 2024-03-29T02:04:40.002 INFO:teuthology.orchestra.run.smithi042.stdout:Release : 68.el9 2024-03-29T02:04:40.002 INFO:teuthology.orchestra.run.smithi042.stdout:Architecture: x86_64 2024-03-29T02:04:40.002 INFO:teuthology.orchestra.run.smithi042.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T02:04:40.003 INFO:teuthology.orchestra.run.smithi042.stdout:Group : Unspecified 2024-03-29T02:04:40.003 INFO:teuthology.orchestra.run.smithi042.stdout:Size : 8264254 2024-03-29T02:04:40.003 INFO:teuthology.orchestra.run.smithi042.stdout:License : GPLv3+ 2024-03-29T02:04:40.003 INFO:teuthology.orchestra.run.smithi042.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T02:04:40.003 INFO:teuthology.orchestra.run.smithi042.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T02:04:40.003 INFO:teuthology.orchestra.run.smithi042.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T02:04:40.003 INFO:teuthology.orchestra.run.smithi042.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T02:04:40.003 INFO:teuthology.orchestra.run.smithi042.stdout:Packager : builder@centos.org 2024-03-29T02:04:40.003 INFO:teuthology.orchestra.run.smithi042.stdout:Vendor : CentOS 2024-03-29T02:04:40.003 INFO:teuthology.orchestra.run.smithi042.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T02:04:40.003 INFO:teuthology.orchestra.run.smithi042.stdout:Summary : Support tools for GRUB. 2024-03-29T02:04:40.003 INFO:teuthology.orchestra.run.smithi042.stdout:Description : 2024-03-29T02:04:40.003 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:04:40.003 INFO:teuthology.orchestra.run.smithi042.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T02:04:40.004 INFO:teuthology.orchestra.run.smithi042.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T02:04:40.004 INFO:teuthology.orchestra.run.smithi042.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T02:04:40.004 INFO:teuthology.orchestra.run.smithi042.stdout:hardware devices. 2024-03-29T02:04:40.004 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:04:40.004 INFO:teuthology.orchestra.run.smithi042.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T02:04:40.005 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T02:04:40.005 INFO:teuthology.task.kernel:Updating grub on smithi042 to boot 5.14.0-434.el9.x86_64 2024-03-29T02:04:40.005 DEBUG:teuthology.orchestra.run.smithi042:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T02:04:40.665 INFO:teuthology.orchestra.run.smithi042.stderr:Generating grub configuration file ... 2024-03-29T02:04:42.081 INFO:teuthology.orchestra.run.smithi042.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T02:04:42.115 INFO:teuthology.orchestra.run.smithi042.stderr:done 2024-03-29T02:04:42.118 DEBUG:teuthology.orchestra.run.smithi042:> mktemp 2024-03-29T02:04:42.174 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.37TY39X2TR 2024-03-29T02:04:42.174 DEBUG:teuthology.orchestra.run.smithi042:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.37TY39X2TR 2024-03-29T02:04:42.246 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.37TY39X2TR 2024-03-29T02:04:42.388 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.37TY39X2TR is 6KB 2024-03-29T02:04:42.437 DEBUG:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.37TY39X2TR 2024-03-29T02:04:42.452 DEBUG:teuthology.orchestra.run.smithi042:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T02:04:42.520 INFO:teuthology.orchestra.run.smithi042.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T02:04:42.521 INFO:teuthology.orchestra.run.smithi042.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T02:04:42.521 INFO:teuthology.orchestra.run.smithi042.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T02:04:42.521 INFO:teuthology.orchestra.run.smithi042.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T02:04:42.522 DEBUG:teuthology.orchestra.run.smithi042:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T02:04:42.677 DEBUG:teuthology.orchestra.run.smithi042:> sudo shutdown -r now 2024-03-29T02:05:12.707 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T02:05:12.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-03-29T02:05:12.708 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T02:06:12.771 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T02:06:21.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-03-29T02:06:21.778 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T02:06:22.186 DEBUG:teuthology.orchestra.run.smithi042:> true 2024-03-29T02:06:22.560 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-03-29T02:06:22.560 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T02:06:22.561 DEBUG:teuthology.orchestra.run.smithi042:> uname -r 2024-03-29T02:06:22.574 INFO:teuthology.orchestra.run.smithi042.stdout:5.14.0-434.el9.x86_64 2024-03-29T02:06:22.575 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T02:06:22.575 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T02:06:22.575 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T02:06:23.575 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T02:06:23.576 DEBUG:teuthology.orchestra.run.smithi042:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T02:06:23.685 INFO:teuthology.orchestra.run.smithi042.stdout:ttyS1 2024-03-29T02:06:23.725 DEBUG:teuthology.parallel:result is None 2024-03-29T02:06:23.725 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T02:06:23.733 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T02:06:23.733 DEBUG:teuthology.orchestra.run.smithi042:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T02:06:23.747 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T02:06:23.753 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T02:06:23.764 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T02:06:23.765 DEBUG:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T02:06:23.811 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T02:06:23.817 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T02:06:23.818 DEBUG:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T02:06:23.874 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T02:06:23.886 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T02:06:23.888 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T02:06:23.895 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T02:06:23.895 DEBUG:teuthology.orchestra.run.smithi042:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T02:06:23.969 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T02:06:23.979 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T02:06:23.979 DEBUG:teuthology.orchestra.run.smithi042:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T02:06:24.023 DEBUG:teuthology.orchestra.run.smithi042:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T02:06:24.105 DEBUG:teuthology.orchestra.run.smithi042:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T02:06:24.181 DEBUG:teuthology.orchestra.run.smithi042:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T02:06:24.215 DEBUG:teuthology.orchestra.run.smithi042:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T02:06:24.263 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:06:24.264 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T02:06:24.330 DEBUG:teuthology.orchestra.run.smithi042:> sudo service rsyslog restart 2024-03-29T02:06:24.539 INFO:teuthology.orchestra.run.smithi042.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T02:06:24.586 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T02:06:24.593 INFO:teuthology.task.internal:Starting timer... 2024-03-29T02:06:24.593 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T02:06:24.628 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T02:06:24.638 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-29T02:06:24.638 DEBUG:teuthology.orchestra.run.smithi042:> sudo service auditd rotate 2024-03-29T02:06:24.725 INFO:teuthology.orchestra.run.smithi042.stdout:Rotating logs: 2024-03-29T02:06:24.732 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T02:06:24.732 DEBUG:teuthology.orchestra.run.smithi042:> /usr/sbin/getenforce 2024-03-29T02:06:24.773 INFO:teuthology.orchestra.run.smithi042.stdout:Permissive 2024-03-29T02:06:24.773 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi042.front.sepia.ceph.com': 'permissive'} 2024-03-29T02:06:24.774 DEBUG:teuthology.orchestra.run.smithi042:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-29T02:06:24.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T02:06:24.867 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T02:06:24.867 DEBUG:teuthology.orchestra.run.smithi042:> sudo /usr/sbin/setenforce permissive 2024-03-29T02:06:24.959 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T02:06:24.970 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T02:06:24.987 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T02:06:24.989 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi042.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T02:10:45.729 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi042.front.sepia.ceph.com')] 2024-03-29T02:10:45.730 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-03-29T02:10:45.732 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi042.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T02:10:45.827 DEBUG:teuthology.orchestra.run.smithi042:> true 2024-03-29T02:10:45.913 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi042.front.sepia.ceph.com' 2024-03-29T02:10:45.913 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T02:10:45.925 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T02:10:45.925 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T02:10:45.926 DEBUG:teuthology.orchestra.run.smithi042:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T02:10:45.986 INFO:teuthology.orchestra.run.smithi042.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T02:10:46.003 INFO:teuthology.orchestra.run.smithi042.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T02:10:46.033 INFO:teuthology.orchestra.run.smithi042.stderr:sudo: ntpd: command not found 2024-03-29T02:10:46.047 INFO:teuthology.orchestra.run.smithi042.stdout:506 Cannot talk to daemon 2024-03-29T02:10:46.064 INFO:teuthology.orchestra.run.smithi042.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T02:10:46.081 INFO:teuthology.orchestra.run.smithi042.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T02:10:46.134 INFO:teuthology.orchestra.run.smithi042.stderr:bash: line 1: ntpq: command not found 2024-03-29T02:10:46.137 INFO:teuthology.orchestra.run.smithi042.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T02:10:46.137 INFO:teuthology.orchestra.run.smithi042.stdout:=============================================================================== 2024-03-29T02:10:46.137 INFO:teuthology.orchestra.run.smithi042.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T02:10:46.137 INFO:teuthology.orchestra.run.smithi042.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T02:10:46.137 INFO:teuthology.orchestra.run.smithi042.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T02:10:46.137 INFO:teuthology.orchestra.run.smithi042.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T02:10:46.138 INFO:teuthology.run_tasks:Running task install... 2024-03-29T02:10:46.146 DEBUG:teuthology.task.install:project ceph 2024-03-29T02:10:46.146 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'}} 2024-03-29T02:10:46.147 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'} 2024-03-29T02:10:46.147 INFO:teuthology.task.install:Using flavor: crimson 2024-03-29T02:10:46.152 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T02:10:46.153 INFO:teuthology.task.install:extra packages: [] 2024-03-29T02:10:46.153 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17', 'tag': None, 'wait_for_package': False} 2024-03-29T02:10:46.153 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=785fc5e8d093374f751c7da3e51e9b56b3718d17 2024-03-29T02:10:46.374 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-3/785fc5e8d093374f751c7da3e51e9b56b3718d17/centos/9/flavors/crimson/ 2024-03-29T02:10:46.374 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2473.g785fc5e8 2024-03-29T02:10:46.485 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-3/785fc5e8d093374f751c7da3e51e9b56b3718d17/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-3/785fc5e8d093374f751c7da3e51e9b56b3718d17/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/ci-yingxin-crimson-load-obc-3/785fc5e8d093374f751c7da3e51e9b56b3718d17/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T02:10:46.486 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:10:46.486 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T02:10:46.515 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T02:10:46.515 DEBUG:teuthology.orchestra.run.smithi042:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/785fc5e8d093374f751c7da3e51e9b56b3718d17/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T02:10:46.584 DEBUG:teuthology.orchestra.run.smithi042:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T02:10:46.666 DEBUG:teuthology.orchestra.run.smithi042:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T02:10:46.733 INFO:teuthology.orchestra.run.smithi042.stdout:check_obsoletes = 1 2024-03-29T02:10:46.734 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum clean all 2024-03-29T02:10:46.989 INFO:teuthology.orchestra.run.smithi042.stdout:74 files removed 2024-03-29T02:10:47.017 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-radosgw 2024-03-29T02:10:48.112 INFO:teuthology.orchestra.run.smithi042.stdout:ceph packages for x86_64 110 kB/s | 87 kB 00:00 2024-03-29T02:10:48.837 INFO:teuthology.orchestra.run.smithi042.stdout:ceph noarch packages 17 kB/s | 11 kB 00:00 2024-03-29T02:10:49.488 INFO:teuthology.orchestra.run.smithi042.stdout:ceph source packages 3.3 kB/s | 2.0 kB 00:00 2024-03-29T02:10:50.099 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-29T02:10:51.574 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-03-29T02:10:52.923 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 9 - BaseOS 7.9 MB/s | 8.0 MB 00:01 2024-03-29T02:10:56.024 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 9 - AppStream 9.6 MB/s | 19 MB 00:01 2024-03-29T02:11:03.076 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 9 - CRB 1.8 MB/s | 6.1 MB 00:03 2024-03-29T02:11:04.685 INFO:teuthology.orchestra.run.smithi042.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 16 kB 00:00 2024-03-29T02:11:24.887 INFO:teuthology.orchestra.run.smithi042.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:20 2024-03-29T02:11:30.010 INFO:teuthology.orchestra.run.smithi042.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-29T02:11:32.750 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:11:32.753 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:11:32.754 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repository Size 2024-03-29T02:11:32.754 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:11:32.754 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-29T02:11:32.754 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-radosgw x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 9.6 M 2024-03-29T02:11:32.754 INFO:teuthology.orchestra.run.smithi042.stdout:Upgrading: 2024-03-29T02:11:32.754 INFO:teuthology.orchestra.run.smithi042.stdout: librados2 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 3.4 M 2024-03-29T02:11:32.754 INFO:teuthology.orchestra.run.smithi042.stdout: librbd1 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 3.7 M 2024-03-29T02:11:32.754 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-03-29T02:11:32.754 INFO:teuthology.orchestra.run.smithi042.stdout: boost-program-options 2024-03-29T02:11:32.754 INFO:teuthology.orchestra.run.smithi042.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T02:11:32.754 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-base x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 5.9 M 2024-03-29T02:11:32.755 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-common x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 19 M 2024-03-29T02:11:32.755 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-selinux x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 25 k 2024-03-29T02:11:32.755 INFO:teuthology.orchestra.run.smithi042.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T02:11:32.755 INFO:teuthology.orchestra.run.smithi042.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T02:11:32.755 INFO:teuthology.orchestra.run.smithi042.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T02:11:32.755 INFO:teuthology.orchestra.run.smithi042.stdout: libcephfs2 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 796 k 2024-03-29T02:11:32.755 INFO:teuthology.orchestra.run.smithi042.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T02:11:32.755 INFO:teuthology.orchestra.run.smithi042.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T02:11:32.755 INFO:teuthology.orchestra.run.smithi042.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T02:11:32.755 INFO:teuthology.orchestra.run.smithi042.stdout: libradosstriper1 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 547 k 2024-03-29T02:11:32.755 INFO:teuthology.orchestra.run.smithi042.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T02:11:32.755 INFO:teuthology.orchestra.run.smithi042.stdout: librgw2 x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 5.0 M 2024-03-29T02:11:32.755 INFO:teuthology.orchestra.run.smithi042.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T02:11:32.756 INFO:teuthology.orchestra.run.smithi042.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T02:11:32.756 INFO:teuthology.orchestra.run.smithi042.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T02:11:32.756 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-argparse 2024-03-29T02:11:32.756 INFO:teuthology.orchestra.run.smithi042.stdout: x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 45 k 2024-03-29T02:11:32.756 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-common x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 134 k 2024-03-29T02:11:32.756 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cephfs x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 159 k 2024-03-29T02:11:32.756 INFO:teuthology.orchestra.run.smithi042.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T02:11:32.756 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rados x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 315 k 2024-03-29T02:11:32.756 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rbd x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 293 k 2024-03-29T02:11:32.756 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rgw x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 99 k 2024-03-29T02:11:32.756 INFO:teuthology.orchestra.run.smithi042.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T02:11:32.756 INFO:teuthology.orchestra.run.smithi042.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T02:11:32.756 INFO:teuthology.orchestra.run.smithi042.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T02:11:32.757 INFO:teuthology.orchestra.run.smithi042.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T02:11:32.757 INFO:teuthology.orchestra.run.smithi042.stdout:Installing weak dependencies: 2024-03-29T02:11:32.757 INFO:teuthology.orchestra.run.smithi042.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T02:11:32.757 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:11:32.757 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-29T02:11:32.757 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:11:32.757 INFO:teuthology.orchestra.run.smithi042.stdout:Install 30 Packages 2024-03-29T02:11:32.757 INFO:teuthology.orchestra.run.smithi042.stdout:Upgrade 2 Packages 2024-03-29T02:11:32.757 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:11:32.758 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 58 M 2024-03-29T02:11:32.758 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-29T02:11:34.777 INFO:teuthology.orchestra.run.smithi042.stdout:(1/32): ceph-base-19.0.0-2473.g785fc5e8.el9.x86 3.6 MB/s | 5.9 MB 00:01 2024-03-29T02:11:34.869 INFO:teuthology.orchestra.run.smithi042.stdout:(2/32): ceph-selinux-19.0.0-2473.g785fc5e8.el9. 274 kB/s | 25 kB 00:00 2024-03-29T02:11:35.194 INFO:teuthology.orchestra.run.smithi042.stdout:(3/32): libcephfs2-19.0.0-2473.g785fc5e8.el9.x8 2.4 MB/s | 796 kB 00:00 2024-03-29T02:11:35.436 INFO:teuthology.orchestra.run.smithi042.stdout:(4/32): libradosstriper1-19.0.0-2473.g785fc5e8. 2.2 MB/s | 547 kB 00:00 2024-03-29T02:11:36.945 INFO:teuthology.orchestra.run.smithi042.stdout:(5/32): ceph-common-19.0.0-2473.g785fc5e8.el9.x 5.0 MB/s | 19 MB 00:03 2024-03-29T02:11:37.028 INFO:teuthology.orchestra.run.smithi042.stdout:(6/32): python3-ceph-argparse-19.0.0-2473.g785f 544 kB/s | 45 kB 00:00 2024-03-29T02:11:37.128 INFO:teuthology.orchestra.run.smithi042.stdout:(7/32): python3-ceph-common-19.0.0-2473.g785fc5 1.3 MB/s | 134 kB 00:00 2024-03-29T02:11:37.304 INFO:teuthology.orchestra.run.smithi042.stdout:(8/32): ceph-radosgw-19.0.0-2473.g785fc5e8.el9. 2.3 MB/s | 9.6 MB 00:04 2024-03-29T02:11:37.329 INFO:teuthology.orchestra.run.smithi042.stdout:(9/32): python3-cephfs-19.0.0-2473.g785fc5e8.el 794 kB/s | 159 kB 00:00 2024-03-29T02:11:37.455 INFO:teuthology.orchestra.run.smithi042.stdout:(10/32): python3-rados-19.0.0-2473.g785fc5e8.el 2.0 MB/s | 315 kB 00:00 2024-03-29T02:11:37.480 INFO:teuthology.orchestra.run.smithi042.stdout:(11/32): python3-rbd-19.0.0-2473.g785fc5e8.el9. 1.9 MB/s | 293 kB 00:00 2024-03-29T02:11:37.539 INFO:teuthology.orchestra.run.smithi042.stdout:(12/32): python3-rgw-19.0.0-2473.g785fc5e8.el9. 1.2 MB/s | 99 kB 00:00 2024-03-29T02:11:37.614 INFO:teuthology.orchestra.run.smithi042.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 159 kB 00:00 2024-03-29T02:11:37.639 INFO:teuthology.orchestra.run.smithi042.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 666 kB/s | 106 kB 00:00 2024-03-29T02:11:37.664 INFO:teuthology.orchestra.run.smithi042.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 887 kB/s | 44 kB 00:00 2024-03-29T02:11:37.698 INFO:teuthology.orchestra.run.smithi042.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-29T02:11:37.723 INFO:teuthology.orchestra.run.smithi042.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-03-29T02:11:37.807 INFO:teuthology.orchestra.run.smithi042.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 381 kB/s | 41 kB 00:00 2024-03-29T02:11:37.841 INFO:teuthology.orchestra.run.smithi042.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.8 MB/s | 220 kB 00:00 2024-03-29T02:11:37.866 INFO:teuthology.orchestra.run.smithi042.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 598 kB/s | 35 kB 00:00 2024-03-29T02:11:37.891 INFO:teuthology.orchestra.run.smithi042.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 923 kB/s | 46 kB 00:00 2024-03-29T02:11:37.958 INFO:teuthology.orchestra.run.smithi042.stdout:(22/32): librgw2-19.0.0-2473.g785fc5e8.el9.x86_ 2.0 MB/s | 5.0 MB 00:02 2024-03-29T02:11:38.025 INFO:teuthology.orchestra.run.smithi042.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 391 kB/s | 26 kB 00:00 2024-03-29T02:11:38.059 INFO:teuthology.orchestra.run.smithi042.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-03-29T02:11:38.084 INFO:teuthology.orchestra.run.smithi042.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 831 kB/s | 49 kB 00:00 2024-03-29T02:11:38.109 INFO:teuthology.orchestra.run.smithi042.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-29T02:11:38.143 INFO:teuthology.orchestra.run.smithi042.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 5.6 MB/s | 192 kB 00:00 2024-03-29T02:11:38.210 INFO:teuthology.orchestra.run.smithi042.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.5 MB/s | 838 kB 00:00 2024-03-29T02:11:38.285 INFO:teuthology.orchestra.run.smithi042.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-03-29T02:11:38.327 INFO:teuthology.orchestra.run.smithi042.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-03-29T02:11:39.903 INFO:teuthology.orchestra.run.smithi042.stdout:(31/32): librbd1-19.0.0-2473.g785fc5e8.el9.x86_ 2.3 MB/s | 3.7 MB 00:01 2024-03-29T02:11:40.269 INFO:teuthology.orchestra.run.smithi042.stdout:(32/32): librados2-19.0.0-2473.g785fc5e8.el9.x8 1.6 MB/s | 3.4 MB 00:02 2024-03-29T02:11:40.273 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:11:40.273 INFO:teuthology.orchestra.run.smithi042.stdout:Total 7.7 MB/s | 58 MB 00:07 2024-03-29T02:11:40.273 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-29T02:11:40.317 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-29T02:11:40.317 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-29T02:11:40.823 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-29T02:11:40.823 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-29T02:11:42.331 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-29T02:11:42.886 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T02:11:42.946 INFO:teuthology.orchestra.run.smithi042.stdout: Upgrading : librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/34 2024-03-29T02:11:43.030 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/34 2024-03-29T02:11:43.122 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-rados-2:19.0.0-2473.g785fc5e8.el9.x86_64 3/34 2024-03-29T02:11:43.148 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 4/34 2024-03-29T02:11:43.236 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 4/34 2024-03-29T02:11:43.277 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-03-29T02:11:43.311 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-03-29T02:11:43.336 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-03-29T02:11:43.359 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-03-29T02:11:43.374 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-03-29T02:11:43.509 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-03-29T02:11:43.541 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-03-29T02:11:43.910 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-03-29T02:11:43.946 INFO:teuthology.orchestra.run.smithi042.stdout: Upgrading : librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/34 2024-03-29T02:11:44.013 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/34 2024-03-29T02:11:44.064 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-ceph-argparse-2:19.0.0-2473.g785fc5e8.el9. 12/34 2024-03-29T02:11:44.120 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cephfs-2:19.0.0-2473.g785fc5e8.el9.x86_64 13/34 2024-03-29T02:11:44.152 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-rbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 14/34 2024-03-29T02:11:44.232 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-03-29T02:11:44.265 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libradosstriper1-2:19.0.0-2473.g785fc5e8.el9.x86_6 16/34 2024-03-29T02:11:44.354 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2473.g785fc5e8.el9.x86_6 16/34 2024-03-29T02:11:44.389 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-03-29T02:11:44.499 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-03-29T02:11:44.548 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-03-29T02:11:44.598 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-03-29T02:11:44.638 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-03-29T02:11:44.667 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-03-29T02:11:44.703 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-03-29T02:11:44.751 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T02:11:45.321 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T02:11:45.355 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : librgw2-2:19.0.0-2473.g785fc5e8.el9.x86_64 26/34 2024-03-29T02:11:45.433 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librgw2-2:19.0.0-2473.g785fc5e8.el9.x86_64 26/34 2024-03-29T02:11:45.505 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-rgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 27/34 2024-03-29T02:11:45.549 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-ceph-common-2:19.0.0-2473.g785fc5e8.el9.x8 28/34 2024-03-29T02:11:48.080 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 29/34 2024-03-29T02:11:48.112 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 29/34 2024-03-29T02:11:48.819 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 29/34 2024-03-29T02:11:48.870 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 30/34 2024-03-29T02:11:48.940 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 30/34 2024-03-29T02:11:48.940 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T02:11:48.940 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T02:11:48.940 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:11:48.975 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-selinux-2:19.0.0-2473.g785fc5e8.el9.x86_64 31/34 2024-03-29T02:11:59.435 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2473.g785fc5e8.el9.x86_64 31/34 2024-03-29T02:11:59.435 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /sys 2024-03-29T02:11:59.435 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /proc 2024-03-29T02:11:59.435 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /mnt 2024-03-29T02:11:59.435 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /var/tmp 2024-03-29T02:11:59.435 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /home 2024-03-29T02:11:59.436 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /root 2024-03-29T02:11:59.436 INFO:teuthology.orchestra.run.smithi042.stdout:skipping the directory /tmp 2024-03-29T02:11:59.436 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:12:00.483 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-radosgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 32/34 2024-03-29T02:12:00.508 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 32/34 2024-03-29T02:12:00.508 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T02:12:00.508 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T02:12:00.508 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T02:12:00.508 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T02:12:00.508 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:12:00.510 INFO:teuthology.orchestra.run.smithi042.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T02:12:00.603 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T02:12:00.603 INFO:teuthology.orchestra.run.smithi042.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T02:12:01.793 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T02:12:01.793 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/34 2024-03-29T02:12:01.793 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/34 2024-03-29T02:12:01.793 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-radosgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 3/34 2024-03-29T02:12:01.793 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-selinux-2:19.0.0-2473.g785fc5e8.el9.x86_64 4/34 2024-03-29T02:12:01.793 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 5/34 2024-03-29T02:12:01.793 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libradosstriper1-2:19.0.0-2473.g785fc5e8.el9.x86_6 6/34 2024-03-29T02:12:01.793 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librgw2-2:19.0.0-2473.g785fc5e8.el9.x86_64 7/34 2024-03-29T02:12:01.793 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2473.g785fc5e8.el9. 8/34 2024-03-29T02:12:01.793 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-ceph-common-2:19.0.0-2473.g785fc5e8.el9.x8 9/34 2024-03-29T02:12:01.793 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cephfs-2:19.0.0-2473.g785fc5e8.el9.x86_64 10/34 2024-03-29T02:12:01.794 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-rados-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/34 2024-03-29T02:12:01.794 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-rbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 12/34 2024-03-29T02:12:01.794 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-rgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 13/34 2024-03-29T02:12:01.794 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T02:12:01.794 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T02:12:01.794 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T02:12:01.794 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T02:12:01.794 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T02:12:01.794 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T02:12:01.794 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T02:12:01.794 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T02:12:01.794 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T02:12:01.794 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T02:12:01.795 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T02:12:01.795 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T02:12:01.795 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T02:12:01.795 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T02:12:01.796 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T02:12:01.796 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T02:12:01.796 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T02:12:01.796 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 31/34 2024-03-29T02:12:01.796 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T02:12:01.796 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 33/34 2024-03-29T02:12:02.864 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T02:12:02.864 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:12:02.864 INFO:teuthology.orchestra.run.smithi042.stdout:Upgraded: 2024-03-29T02:12:02.864 INFO:teuthology.orchestra.run.smithi042.stdout: librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:12:02.864 INFO:teuthology.orchestra.run.smithi042.stdout: librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:12:02.864 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-29T02:12:02.864 INFO:teuthology.orchestra.run.smithi042.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T02:12:02.864 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:12:02.865 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:12:02.865 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-radosgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:12:02.865 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-selinux-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:12:02.865 INFO:teuthology.orchestra.run.smithi042.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T02:12:02.865 INFO:teuthology.orchestra.run.smithi042.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T02:12:02.865 INFO:teuthology.orchestra.run.smithi042.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T02:12:02.865 INFO:teuthology.orchestra.run.smithi042.stdout: libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:12:02.865 INFO:teuthology.orchestra.run.smithi042.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T02:12:02.865 INFO:teuthology.orchestra.run.smithi042.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T02:12:02.865 INFO:teuthology.orchestra.run.smithi042.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T02:12:02.865 INFO:teuthology.orchestra.run.smithi042.stdout: libradosstriper1-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:12:02.865 INFO:teuthology.orchestra.run.smithi042.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T02:12:02.865 INFO:teuthology.orchestra.run.smithi042.stdout: librgw2-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:12:02.865 INFO:teuthology.orchestra.run.smithi042.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T02:12:02.866 INFO:teuthology.orchestra.run.smithi042.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T02:12:02.866 INFO:teuthology.orchestra.run.smithi042.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T02:12:02.866 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-argparse-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:12:02.866 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ceph-common-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:12:02.866 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cephfs-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:12:02.866 INFO:teuthology.orchestra.run.smithi042.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T02:12:02.866 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rados-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:12:02.866 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:12:02.866 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:12:02.866 INFO:teuthology.orchestra.run.smithi042.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T02:12:02.866 INFO:teuthology.orchestra.run.smithi042.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T02:12:02.866 INFO:teuthology.orchestra.run.smithi042.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T02:12:02.866 INFO:teuthology.orchestra.run.smithi042.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T02:12:02.867 INFO:teuthology.orchestra.run.smithi042.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T02:12:02.867 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:12:02.867 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:12:03.158 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-test 2024-03-29T02:12:03.720 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:00:34 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:12:03.879 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:12:03.880 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:12:03.880 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repository Size 2024-03-29T02:12:03.880 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:12:03.880 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-29T02:12:03.880 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-test x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 87 M 2024-03-29T02:12:03.880 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-03-29T02:12:03.880 INFO:teuthology.orchestra.run.smithi042.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T02:12:03.880 INFO:teuthology.orchestra.run.smithi042.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-03-29T02:12:03.880 INFO:teuthology.orchestra.run.smithi042.stdout: libcephsqlite x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 174 k 2024-03-29T02:12:03.880 INFO:teuthology.orchestra.run.smithi042.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-03-29T02:12:03.881 INFO:teuthology.orchestra.run.smithi042.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2024-03-29T02:12:03.881 INFO:teuthology.orchestra.run.smithi042.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T02:12:03.881 INFO:teuthology.orchestra.run.smithi042.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T02:12:03.881 INFO:teuthology.orchestra.run.smithi042.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T02:12:03.881 INFO:teuthology.orchestra.run.smithi042.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T02:12:03.881 INFO:teuthology.orchestra.run.smithi042.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-03-29T02:12:03.881 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:12:03.881 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-29T02:12:03.881 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:12:03.881 INFO:teuthology.orchestra.run.smithi042.stdout:Install 11 Packages 2024-03-29T02:12:03.881 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:12:03.882 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 90 M 2024-03-29T02:12:03.882 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 819 M 2024-03-29T02:12:03.882 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-29T02:12:04.508 INFO:teuthology.orchestra.run.smithi042.stdout:(1/11): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T02:12:04.642 INFO:teuthology.orchestra.run.smithi042.stdout:(2/11): protobuf-3.14.0-13.el9.x86_64.rpm 7.7 MB/s | 1.0 MB 00:00 2024-03-29T02:12:04.667 INFO:teuthology.orchestra.run.smithi042.stdout:(3/11): libcephsqlite-19.0.0-2473.g785fc5e8.el9 542 kB/s | 174 kB 00:00 2024-03-29T02:12:04.701 INFO:teuthology.orchestra.run.smithi042.stdout:(4/11): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-29T02:12:04.717 INFO:teuthology.orchestra.run.smithi042.stdout:(5/11): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-29T02:12:04.743 INFO:teuthology.orchestra.run.smithi042.stdout:(6/11): lksctp-tools-1.0.19-2.el9.x86_64.rpm 2.1 MB/s | 90 kB 00:00 2024-03-29T02:12:04.952 INFO:teuthology.orchestra.run.smithi042.stdout:(7/11): jq-1.6-16.el9.x86_64.rpm 801 kB/s | 187 kB 00:00 2024-03-29T02:12:05.202 INFO:teuthology.orchestra.run.smithi042.stdout:(8/11): libasan8-13.2.1-6.3.el9.x86_64.rpm 1.0 MB/s | 469 kB 00:00 2024-03-29T02:12:05.336 INFO:teuthology.orchestra.run.smithi042.stdout:(9/11): libubsan-11.4.1-3.el9.x86_64.rpm 487 kB/s | 187 kB 00:00 2024-03-29T02:12:05.636 INFO:teuthology.orchestra.run.smithi042.stdout:(10/11): yaml-cpp-0.6.3-5.el9.x86_64.rpm 281 kB/s | 122 kB 00:00 2024-03-29T02:12:19.094 INFO:teuthology.orchestra.run.smithi042.stdout:(11/11): ceph-test-19.0.0-2473.g785fc5e8.el9.x8 5.9 MB/s | 87 MB 00:14 2024-03-29T02:12:19.096 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:12:19.096 INFO:teuthology.orchestra.run.smithi042.stdout:Total 5.9 MB/s | 90 MB 00:15 2024-03-29T02:12:19.137 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-29T02:12:19.163 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-29T02:12:19.163 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-29T02:12:19.821 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-29T02:12:19.821 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-29T02:12:20.859 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-29T02:12:20.923 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/11 2024-03-29T02:12:20.953 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 2/11 2024-03-29T02:12:20.978 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-03-29T02:12:21.062 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 3/11 2024-03-29T02:12:21.095 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 4/11 2024-03-29T02:12:21.139 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/11 2024-03-29T02:12:21.193 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/11 2024-03-29T02:12:21.224 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 7/11 2024-03-29T02:12:21.263 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 8/11 2024-03-29T02:12:21.308 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : jq-1.6-16.el9.x86_64 9/11 2024-03-29T02:12:21.334 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libcephsqlite-2:19.0.0-2473.g785fc5e8.el9.x86_64 10/11 2024-03-29T02:12:33.506 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2473.g785fc5e8.el9.x86_64 10/11 2024-03-29T02:12:33.585 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-test-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/11 2024-03-29T02:12:39.787 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-test-2:19.0.0-2473.g785fc5e8.el9.x86_64 11/11 2024-03-29T02:12:39.787 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-test-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/11 2024-03-29T02:12:39.787 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libcephsqlite-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/11 2024-03-29T02:12:39.787 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/11 2024-03-29T02:12:39.787 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/11 2024-03-29T02:12:39.787 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/11 2024-03-29T02:12:39.787 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/11 2024-03-29T02:12:39.787 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 7/11 2024-03-29T02:12:39.787 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : jq-1.6-16.el9.x86_64 8/11 2024-03-29T02:12:39.787 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 9/11 2024-03-29T02:12:39.788 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 10/11 2024-03-29T02:12:40.234 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 11/11 2024-03-29T02:12:40.234 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:12:40.234 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-29T02:12:40.235 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-test-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:12:40.235 INFO:teuthology.orchestra.run.smithi042.stdout: jq-1.6-16.el9.x86_64 2024-03-29T02:12:40.235 INFO:teuthology.orchestra.run.smithi042.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-03-29T02:12:40.235 INFO:teuthology.orchestra.run.smithi042.stdout: libcephsqlite-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:12:40.235 INFO:teuthology.orchestra.run.smithi042.stdout: libubsan-11.4.1-3.el9.x86_64 2024-03-29T02:12:40.235 INFO:teuthology.orchestra.run.smithi042.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-03-29T02:12:40.235 INFO:teuthology.orchestra.run.smithi042.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T02:12:40.235 INFO:teuthology.orchestra.run.smithi042.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T02:12:40.235 INFO:teuthology.orchestra.run.smithi042.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T02:12:40.235 INFO:teuthology.orchestra.run.smithi042.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T02:12:40.235 INFO:teuthology.orchestra.run.smithi042.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-03-29T02:12:40.235 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:12:40.235 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:12:40.522 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph 2024-03-29T02:12:41.098 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:01:12 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:12:41.258 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:12:41.260 INFO:teuthology.orchestra.run.smithi042.stdout:========================================================================================= 2024-03-29T02:12:41.260 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repository Size 2024-03-29T02:12:41.260 INFO:teuthology.orchestra.run.smithi042.stdout:========================================================================================= 2024-03-29T02:12:41.260 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-29T02:12:41.261 INFO:teuthology.orchestra.run.smithi042.stdout: ceph x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 6.5 k 2024-03-29T02:12:41.261 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-03-29T02:12:41.261 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mds x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 2.2 M 2024-03-29T02:12:41.261 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 1.6 M 2024-03-29T02:12:41.261 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 249 k 2024-03-29T02:12:41.261 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mon x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 5.0 M 2024-03-29T02:12:41.261 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-osd x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 71 M 2024-03-29T02:12:41.261 INFO:teuthology.orchestra.run.smithi042.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T02:12:41.261 INFO:teuthology.orchestra.run.smithi042.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T02:12:41.261 INFO:teuthology.orchestra.run.smithi042.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T02:12:41.261 INFO:teuthology.orchestra.run.smithi042.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T02:12:41.261 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T02:12:41.262 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T02:12:41.262 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T02:12:41.262 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T02:12:41.262 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T02:12:41.262 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T02:12:41.262 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T02:12:41.262 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T02:12:41.262 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T02:12:41.262 INFO:teuthology.orchestra.run.smithi042.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T02:12:41.262 INFO:teuthology.orchestra.run.smithi042.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T02:12:41.262 INFO:teuthology.orchestra.run.smithi042.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T02:12:41.262 INFO:teuthology.orchestra.run.smithi042.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T02:12:41.262 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T02:12:41.263 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T02:12:41.263 INFO:teuthology.orchestra.run.smithi042.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T02:12:41.263 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T02:12:41.263 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T02:12:41.263 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T02:12:41.263 INFO:teuthology.orchestra.run.smithi042.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T02:12:41.263 INFO:teuthology.orchestra.run.smithi042.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T02:12:41.263 INFO:teuthology.orchestra.run.smithi042.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T02:12:41.263 INFO:teuthology.orchestra.run.smithi042.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T02:12:41.263 INFO:teuthology.orchestra.run.smithi042.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T02:12:41.263 INFO:teuthology.orchestra.run.smithi042.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T02:12:41.263 INFO:teuthology.orchestra.run.smithi042.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T02:12:41.263 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:12:41.263 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-29T02:12:41.264 INFO:teuthology.orchestra.run.smithi042.stdout:========================================================================================= 2024-03-29T02:12:41.264 INFO:teuthology.orchestra.run.smithi042.stdout:Install 36 Packages 2024-03-29T02:12:41.264 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:12:41.264 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 84 M 2024-03-29T02:12:41.264 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 921 M 2024-03-29T02:12:41.264 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-29T02:12:41.942 INFO:teuthology.orchestra.run.smithi042.stdout:(1/36): ceph-19.0.0-2473.g785fc5e8.el9.x86_64.r 28 kB/s | 6.5 kB 00:00 2024-03-29T02:12:43.001 INFO:teuthology.orchestra.run.smithi042.stdout:(2/36): ceph-mon-19.0.0-2473.g785fc5e8.el9.x86_ 4.7 MB/s | 5.0 MB 00:01 2024-03-29T02:12:43.067 INFO:teuthology.orchestra.run.smithi042.stdout:(3/36): ceph-mgr-19.0.0-2473.g785fc5e8.el9.x86_ 1.1 MB/s | 1.6 MB 00:01 2024-03-29T02:12:43.184 INFO:teuthology.orchestra.run.smithi042.stdout:(4/36): ceph-mds-19.0.0-2473.g785fc5e8.el9.x86_ 1.5 MB/s | 2.2 MB 00:01 2024-03-29T02:12:43.210 INFO:teuthology.orchestra.run.smithi042.stdout:(5/36): ceph-mgr-modules-core-19.0.0-2473.g785f 1.7 MB/s | 249 kB 00:00 2024-03-29T02:12:43.343 INFO:teuthology.orchestra.run.smithi042.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-29T02:12:43.368 INFO:teuthology.orchestra.run.smithi042.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 910 kB/s | 144 kB 00:00 2024-03-29T02:12:43.394 INFO:teuthology.orchestra.run.smithi042.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 630 kB/s | 32 kB 00:00 2024-03-29T02:12:43.419 INFO:teuthology.orchestra.run.smithi042.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 943 kB/s | 47 kB 00:00 2024-03-29T02:12:43.444 INFO:teuthology.orchestra.run.smithi042.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 772 kB/s | 39 kB 00:00 2024-03-29T02:12:43.470 INFO:teuthology.orchestra.run.smithi042.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T02:12:43.520 INFO:teuthology.orchestra.run.smithi042.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T02:12:43.545 INFO:teuthology.orchestra.run.smithi042.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-29T02:12:43.570 INFO:teuthology.orchestra.run.smithi042.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T02:12:43.604 INFO:teuthology.orchestra.run.smithi042.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-03-29T02:12:43.880 INFO:teuthology.orchestra.run.smithi042.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 81 kB/s | 22 kB 00:00 2024-03-29T02:12:44.055 INFO:teuthology.orchestra.run.smithi042.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 865 kB/s | 151 kB 00:00 2024-03-29T02:12:44.113 INFO:teuthology.orchestra.run.smithi042.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 748 kB/s | 43 kB 00:00 2024-03-29T02:12:44.180 INFO:teuthology.orchestra.run.smithi042.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-03-29T02:12:44.255 INFO:teuthology.orchestra.run.smithi042.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 4.7 MB/s | 358 kB 00:00 2024-03-29T02:12:44.315 INFO:teuthology.orchestra.run.smithi042.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-03-29T02:12:44.356 INFO:teuthology.orchestra.run.smithi042.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 425 kB/s | 18 kB 00:00 2024-03-29T02:12:44.382 INFO:teuthology.orchestra.run.smithi042.stdout:(23/36): python3-requests-2.25.1-8.el9.noarch.r 154 kB/s | 125 kB 00:00 2024-03-29T02:12:44.407 INFO:teuthology.orchestra.run.smithi042.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-03-29T02:12:44.449 INFO:teuthology.orchestra.run.smithi042.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 471 kB/s | 20 kB 00:00 2024-03-29T02:12:44.474 INFO:teuthology.orchestra.run.smithi042.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 212 kB/s | 19 kB 00:00 2024-03-29T02:12:44.500 INFO:teuthology.orchestra.run.smithi042.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 924 kB/s | 46 kB 00:00 2024-03-29T02:12:44.558 INFO:teuthology.orchestra.run.smithi042.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.6 MB/s | 272 kB 00:00 2024-03-29T02:12:44.583 INFO:teuthology.orchestra.run.smithi042.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 724 kB/s | 79 kB 00:00 2024-03-29T02:12:44.617 INFO:teuthology.orchestra.run.smithi042.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-03-29T02:12:44.651 INFO:teuthology.orchestra.run.smithi042.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-03-29T02:12:44.676 INFO:teuthology.orchestra.run.smithi042.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 612 kB/s | 36 kB 00:00 2024-03-29T02:12:44.735 INFO:teuthology.orchestra.run.smithi042.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-29T02:12:44.760 INFO:teuthology.orchestra.run.smithi042.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-03-29T02:12:44.785 INFO:teuthology.orchestra.run.smithi042.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-03-29T02:12:46.569 INFO:teuthology.orchestra.run.smithi042.stdout:(36/36): ceph-osd-19.0.0-2473.g785fc5e8.el9.x86 20 MB/s | 71 MB 00:03 2024-03-29T02:12:46.574 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:12:46.574 INFO:teuthology.orchestra.run.smithi042.stdout:Total 16 MB/s | 84 MB 00:05 2024-03-29T02:12:46.599 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-29T02:12:46.646 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-29T02:12:46.647 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-29T02:12:47.400 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-29T02:12:47.400 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-29T02:12:49.016 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-29T02:12:49.071 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T02:12:49.119 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T02:12:49.157 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T02:12:49.232 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T02:12:49.273 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T02:12:49.306 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T02:12:49.410 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T02:12:49.496 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T02:12:49.542 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T02:12:49.615 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T02:12:49.669 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T02:12:49.715 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T02:12:49.773 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T02:12:49.828 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T02:12:50.024 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T02:12:50.077 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T02:12:50.116 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T02:12:50.150 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T02:12:50.180 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T02:12:50.261 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T02:12:50.299 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T02:12:50.332 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T02:12:50.373 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T02:12:50.425 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T02:12:50.467 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T02:12:50.562 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T02:12:50.598 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T02:12:50.681 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T02:12:50.832 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T02:12:50.954 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T02:12:51.210 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2473.g785fc5e8.el9. 31/36 2024-03-29T02:12:51.245 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 32/36 2024-03-29T02:12:51.270 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 32/36 2024-03-29T02:12:51.270 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T02:12:51.270 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T02:12:51.270 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T02:12:51.271 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T02:12:51.271 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:02.686 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-osd-2:19.0.0-2473.g785fc5e8.el9.x86_64 33/36 2024-03-29T02:13:02.712 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-osd-2:19.0.0-2473.g785fc5e8.el9.x86_64 33/36 2024-03-29T02:13:02.712 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T02:13:02.713 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T02:13:02.713 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T02:13:02.713 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T02:13:02.713 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:03.287 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mon-2:19.0.0-2473.g785fc5e8.el9.x86_64 34/36 2024-03-29T02:13:03.311 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mon-2:19.0.0-2473.g785fc5e8.el9.x86_64 34/36 2024-03-29T02:13:03.311 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T02:13:03.311 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T02:13:03.311 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T02:13:03.311 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T02:13:03.311 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:03.563 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mds-2:19.0.0-2473.g785fc5e8.el9.x86_64 35/36 2024-03-29T02:13:03.586 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mds-2:19.0.0-2473.g785fc5e8.el9.x86_64 35/36 2024-03-29T02:13:03.586 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T02:13:03.586 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T02:13:03.586 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T02:13:03.586 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T02:13:03.586 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:03.655 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-2:19.0.0-2473.g785fc5e8.el9.x86_64 36/36 2024-03-29T02:13:10.743 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-2:19.0.0-2473.g785fc5e8.el9.x86_64 36/36 2024-03-29T02:13:10.743 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/36 2024-03-29T02:13:10.743 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mds-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/36 2024-03-29T02:13:10.744 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 3/36 2024-03-29T02:13:10.744 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mon-2:19.0.0-2473.g785fc5e8.el9.x86_64 4/36 2024-03-29T02:13:10.744 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-osd-2:19.0.0-2473.g785fc5e8.el9.x86_64 5/36 2024-03-29T02:13:10.744 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2473.g785fc5e8.el9. 6/36 2024-03-29T02:13:10.744 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T02:13:10.744 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T02:13:10.744 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T02:13:10.744 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T02:13:10.744 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T02:13:10.744 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T02:13:10.744 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T02:13:10.744 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T02:13:10.744 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T02:13:10.745 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T02:13:10.745 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T02:13:10.745 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T02:13:10.745 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T02:13:10.745 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T02:13:10.745 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T02:13:10.745 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T02:13:10.745 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T02:13:10.745 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T02:13:10.745 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T02:13:10.745 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T02:13:10.745 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T02:13:10.745 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T02:13:10.745 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T02:13:10.746 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T02:13:10.746 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T02:13:10.746 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T02:13:10.746 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T02:13:10.746 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T02:13:10.746 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T02:13:11.976 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T02:13:11.976 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:11.976 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-29T02:13:11.977 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:13:11.977 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mds-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:13:11.977 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:13:11.977 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-modules-core-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:13:11.977 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mon-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:13:11.977 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-osd-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:13:11.977 INFO:teuthology.orchestra.run.smithi042.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T02:13:11.977 INFO:teuthology.orchestra.run.smithi042.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T02:13:11.977 INFO:teuthology.orchestra.run.smithi042.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T02:13:11.977 INFO:teuthology.orchestra.run.smithi042.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T02:13:11.977 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T02:13:11.977 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T02:13:11.977 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T02:13:11.978 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T02:13:11.978 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T02:13:11.978 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T02:13:11.978 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T02:13:11.978 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T02:13:11.978 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T02:13:11.978 INFO:teuthology.orchestra.run.smithi042.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T02:13:11.978 INFO:teuthology.orchestra.run.smithi042.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T02:13:11.978 INFO:teuthology.orchestra.run.smithi042.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T02:13:11.978 INFO:teuthology.orchestra.run.smithi042.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T02:13:11.978 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T02:13:11.978 INFO:teuthology.orchestra.run.smithi042.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T02:13:11.979 INFO:teuthology.orchestra.run.smithi042.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T02:13:11.979 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T02:13:11.979 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T02:13:11.979 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T02:13:11.979 INFO:teuthology.orchestra.run.smithi042.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T02:13:11.979 INFO:teuthology.orchestra.run.smithi042.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T02:13:11.979 INFO:teuthology.orchestra.run.smithi042.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T02:13:11.979 INFO:teuthology.orchestra.run.smithi042.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T02:13:11.979 INFO:teuthology.orchestra.run.smithi042.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T02:13:11.979 INFO:teuthology.orchestra.run.smithi042.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T02:13:11.979 INFO:teuthology.orchestra.run.smithi042.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T02:13:11.979 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:11.980 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:13:12.228 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-base 2024-03-29T02:13:12.799 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:13:12.900 INFO:teuthology.orchestra.run.smithi042.stdout:Package ceph-base-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:13:12.957 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:13:12.957 INFO:teuthology.orchestra.run.smithi042.stdout:Nothing to do. 2024-03-29T02:13:12.958 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:13:13.024 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install cephadm 2024-03-29T02:13:13.596 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:13:13.753 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:13:13.753 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:13:13.754 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repository Size 2024-03-29T02:13:13.754 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:13:13.754 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-29T02:13:13.754 INFO:teuthology.orchestra.run.smithi042.stdout: cephadm noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 762 k 2024-03-29T02:13:13.754 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:13.754 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-29T02:13:13.754 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:13:13.754 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-29T02:13:13.754 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:13.754 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 762 k 2024-03-29T02:13:13.754 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 768 k 2024-03-29T02:13:13.754 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-29T02:13:14.421 INFO:teuthology.orchestra.run.smithi042.stdout:cephadm-19.0.0-2473.g785fc5e8.el9.noarch.rpm 1.1 MB/s | 762 kB 00:00 2024-03-29T02:13:14.421 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:13:14.422 INFO:teuthology.orchestra.run.smithi042.stdout:Total 1.1 MB/s | 762 kB 00:00 2024-03-29T02:13:14.422 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-29T02:13:14.427 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-29T02:13:14.427 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-29T02:13:14.433 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-29T02:13:14.434 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-29T02:13:14.505 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-29T02:13:14.973 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/1 2024-03-29T02:13:15.070 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/1 2024-03-29T02:13:15.437 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/1 2024-03-29T02:13:15.797 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/1 2024-03-29T02:13:15.797 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:15.797 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-29T02:13:15.797 INFO:teuthology.orchestra.run.smithi042.stdout: cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:13:15.797 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:15.797 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:13:16.000 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T02:13:16.578 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:13:16.742 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:13:16.742 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:13:16.742 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-29T02:13:16.743 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:13:16.743 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-29T02:13:16.743 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 157 k 2024-03-29T02:13:16.743 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:16.743 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-29T02:13:16.743 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:13:16.743 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-29T02:13:16.743 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:16.743 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 157 k 2024-03-29T02:13:16.743 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 640 k 2024-03-29T02:13:16.743 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-29T02:13:17.093 INFO:teuthology.orchestra.run.smithi042.stdout:ceph-immutable-object-cache-19.0.0-2473.g785fc5 449 kB/s | 157 kB 00:00 2024-03-29T02:13:17.094 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:13:17.094 INFO:teuthology.orchestra.run.smithi042.stdout:Total 447 kB/s | 157 kB 00:00 2024-03-29T02:13:17.094 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-29T02:13:17.104 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-29T02:13:17.105 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-29T02:13:17.160 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-29T02:13:17.160 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-29T02:13:17.310 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-29T02:13:17.386 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2473.g785fc5e8. 1/1 2024-03-29T02:13:17.410 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2473.g785fc5e8. 1/1 2024-03-29T02:13:17.410 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T02:13:17.410 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T02:13:17.410 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:18.411 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2473.g785fc5e8. 1/1 2024-03-29T02:13:18.411 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:18.411 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-29T02:13:18.411 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-immutable-object-cache-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:13:18.411 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:18.411 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:13:18.632 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-mgr 2024-03-29T02:13:19.207 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:13:19.310 INFO:teuthology.orchestra.run.smithi042.stdout:Package ceph-mgr-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:13:19.365 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:13:19.366 INFO:teuthology.orchestra.run.smithi042.stdout:Nothing to do. 2024-03-29T02:13:19.366 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:13:19.433 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T02:13:20.001 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:01:51 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:13:20.158 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:13:20.159 INFO:teuthology.orchestra.run.smithi042.stdout:====================================================================================== 2024-03-29T02:13:20.159 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repository Size 2024-03-29T02:13:20.159 INFO:teuthology.orchestra.run.smithi042.stdout:====================================================================================== 2024-03-29T02:13:20.159 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-29T02:13:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 3.5 M 2024-03-29T02:13:20.160 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-03-29T02:13:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T02:13:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 34 k 2024-03-29T02:13:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 16 k 2024-03-29T02:13:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T02:13:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T02:13:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T02:13:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T02:13:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T02:13:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T02:13:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T02:13:20.160 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:20.160 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-29T02:13:20.161 INFO:teuthology.orchestra.run.smithi042.stdout:====================================================================================== 2024-03-29T02:13:20.161 INFO:teuthology.orchestra.run.smithi042.stdout:Install 11 Packages 2024-03-29T02:13:20.161 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:20.161 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 7.6 M 2024-03-29T02:13:20.161 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 100 M 2024-03-29T02:13:20.161 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-29T02:13:21.072 INFO:teuthology.orchestra.run.smithi042.stdout:(1/11): ceph-grafana-dashboards-19.0.0-2473.g78 130 kB/s | 34 kB 00:00 2024-03-29T02:13:21.122 INFO:teuthology.orchestra.run.smithi042.stdout:(2/11): ceph-prometheus-alerts-19.0.0-2473.g785 53 kB/s | 16 kB 00:00 2024-03-29T02:13:21.189 INFO:teuthology.orchestra.run.smithi042.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 238 kB 00:00 2024-03-29T02:13:21.798 INFO:teuthology.orchestra.run.smithi042.stdout:(4/11): protobuf-compiler-3.14.0-13.el9.x86_64. 1.2 MB/s | 863 kB 00:00 2024-03-29T02:13:21.898 INFO:teuthology.orchestra.run.smithi042.stdout:(5/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 773 kB/s | 548 kB 00:00 2024-03-29T02:13:22.215 INFO:teuthology.orchestra.run.smithi042.stdout:(6/11): ceph-mgr-dashboard-19.0.0-2473.g785fc5e 2.5 MB/s | 3.5 MB 00:01 2024-03-29T02:13:22.231 INFO:teuthology.orchestra.run.smithi042.stdout:(7/11): grpc-data-1.46.7-10.el9.noarch.rpm 45 kB/s | 19 kB 00:00 2024-03-29T02:13:22.282 INFO:teuthology.orchestra.run.smithi042.stdout:(8/11): python3-grpcio-1.46.7-10.el9.x86_64.rpm 5.7 MB/s | 2.0 MB 00:00 2024-03-29T02:13:22.374 INFO:teuthology.orchestra.run.smithi042.stdout:(9/11): python3-repoze-lru-0.7-16.el9.noarch.rp 216 kB/s | 31 kB 00:00 2024-03-29T02:13:22.399 INFO:teuthology.orchestra.run.smithi042.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-03-29T02:13:22.774 INFO:teuthology.orchestra.run.smithi042.stdout:(11/11): python3-grpcio-tools-1.46.7-10.el9.x86 258 kB/s | 144 kB 00:00 2024-03-29T02:13:22.781 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:13:22.781 INFO:teuthology.orchestra.run.smithi042.stdout:Total 2.9 MB/s | 7.6 MB 00:02 2024-03-29T02:13:22.802 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-29T02:13:22.822 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-29T02:13:22.822 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-29T02:13:23.025 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-29T02:13:23.025 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-29T02:13:23.513 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-29T02:13:23.607 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-03-29T02:13:23.644 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-03-29T02:13:23.758 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-03-29T02:13:23.901 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-03-29T02:13:23.957 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-03-29T02:13:24.051 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-03-29T02:13:24.101 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-03-29T02:13:24.128 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-03-29T02:13:24.161 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2473.g785fc5e8.el9 9/11 2024-03-29T02:13:25.269 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2473.g785fc5e8.el 10/11 2024-03-29T02:13:25.311 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2473.g785fc5e8.el9.noa 11/11 2024-03-29T02:13:26.637 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2473.g785fc5e8.el9.noa 11/11 2024-03-29T02:13:26.637 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2473.g785fc5e8.el 1/11 2024-03-29T02:13:26.637 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2473.g785fc5e8.el9.noa 2/11 2024-03-29T02:13:26.638 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2473.g785fc5e8.el9 3/11 2024-03-29T02:13:26.638 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-03-29T02:13:26.638 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-03-29T02:13:26.638 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-03-29T02:13:26.638 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-03-29T02:13:26.638 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-03-29T02:13:26.638 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-03-29T02:13:26.638 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-03-29T02:13:27.002 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-03-29T02:13:27.002 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:27.002 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-29T02:13:27.002 INFO:teuthology.orchestra.run.smithi042.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T02:13:27.002 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-grafana-dashboards-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:13:27.002 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-dashboard-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:13:27.002 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-prometheus-alerts-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:13:27.003 INFO:teuthology.orchestra.run.smithi042.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T02:13:27.003 INFO:teuthology.orchestra.run.smithi042.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T02:13:27.003 INFO:teuthology.orchestra.run.smithi042.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T02:13:27.003 INFO:teuthology.orchestra.run.smithi042.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T02:13:27.003 INFO:teuthology.orchestra.run.smithi042.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T02:13:27.003 INFO:teuthology.orchestra.run.smithi042.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T02:13:27.003 INFO:teuthology.orchestra.run.smithi042.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T02:13:27.003 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:27.003 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:13:27.254 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T02:13:27.823 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:13:27.982 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:13:27.983 INFO:teuthology.orchestra.run.smithi042.stdout:============================================================================================ 2024-03-29T02:13:27.983 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repository Size 2024-03-29T02:13:27.983 INFO:teuthology.orchestra.run.smithi042.stdout:============================================================================================ 2024-03-29T02:13:27.983 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-29T02:13:27.983 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 7.4 M 2024-03-29T02:13:27.983 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-03-29T02:13:27.983 INFO:teuthology.orchestra.run.smithi042.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T02:13:27.983 INFO:teuthology.orchestra.run.smithi042.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T02:13:27.984 INFO:teuthology.orchestra.run.smithi042.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T02:13:27.984 INFO:teuthology.orchestra.run.smithi042.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T02:13:27.984 INFO:teuthology.orchestra.run.smithi042.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T02:13:27.984 INFO:teuthology.orchestra.run.smithi042.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T02:13:27.984 INFO:teuthology.orchestra.run.smithi042.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T02:13:27.984 INFO:teuthology.orchestra.run.smithi042.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T02:13:27.984 INFO:teuthology.orchestra.run.smithi042.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T02:13:27.984 INFO:teuthology.orchestra.run.smithi042.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T02:13:27.984 INFO:teuthology.orchestra.run.smithi042.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T02:13:27.984 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:27.984 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-29T02:13:27.984 INFO:teuthology.orchestra.run.smithi042.stdout:============================================================================================ 2024-03-29T02:13:27.984 INFO:teuthology.orchestra.run.smithi042.stdout:Install 12 Packages 2024-03-29T02:13:27.984 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:27.985 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 37 M 2024-03-29T02:13:27.985 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 209 M 2024-03-29T02:13:27.985 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-29T02:13:28.282 INFO:teuthology.orchestra.run.smithi042.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 255 kB/s | 32 kB 00:00 2024-03-29T02:13:28.316 INFO:teuthology.orchestra.run.smithi042.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 518 kB/s | 17 kB 00:00 2024-03-29T02:13:28.349 INFO:teuthology.orchestra.run.smithi042.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-29T02:13:28.449 INFO:teuthology.orchestra.run.smithi042.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-03-29T02:13:28.575 INFO:teuthology.orchestra.run.smithi042.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-03-29T02:13:28.675 INFO:teuthology.orchestra.run.smithi042.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-29T02:13:28.700 INFO:teuthology.orchestra.run.smithi042.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 217 kB 00:00 2024-03-29T02:13:28.817 INFO:teuthology.orchestra.run.smithi042.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 6.8 MB/s | 803 kB 00:00 2024-03-29T02:13:28.901 INFO:teuthology.orchestra.run.smithi042.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 195 kB 00:00 2024-03-29T02:13:29.076 INFO:teuthology.orchestra.run.smithi042.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-03-29T02:13:29.510 INFO:teuthology.orchestra.run.smithi042.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 405 kB/s | 245 kB 00:00 2024-03-29T02:13:33.193 INFO:teuthology.orchestra.run.smithi042.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-2 1.5 MB/s | 7.4 MB 00:05 2024-03-29T02:13:33.196 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:13:33.196 INFO:teuthology.orchestra.run.smithi042.stdout:Total 7.1 MB/s | 37 MB 00:05 2024-03-29T02:13:33.241 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-29T02:13:33.267 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-29T02:13:33.267 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-29T02:13:33.643 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-29T02:13:33.643 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-29T02:13:34.492 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-29T02:13:34.566 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T02:13:34.606 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T02:13:34.711 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T02:13:34.751 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T02:13:34.933 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T02:13:34.966 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T02:13:35.060 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T02:13:35.742 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T02:13:35.858 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T02:13:37.181 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T02:13:38.248 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T02:13:38.273 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2473.g785fc 12/12 2024-03-29T02:13:40.874 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2473.g785fc 12/12 2024-03-29T02:13:40.875 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2473.g785fc 1/12 2024-03-29T02:13:40.875 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T02:13:40.875 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T02:13:40.875 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T02:13:40.875 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T02:13:40.875 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T02:13:40.875 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T02:13:40.875 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T02:13:40.875 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T02:13:40.875 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T02:13:40.875 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T02:13:41.338 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T02:13:41.338 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:41.338 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-29T02:13:41.338 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:13:41.338 INFO:teuthology.orchestra.run.smithi042.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T02:13:41.338 INFO:teuthology.orchestra.run.smithi042.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T02:13:41.338 INFO:teuthology.orchestra.run.smithi042.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T02:13:41.338 INFO:teuthology.orchestra.run.smithi042.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T02:13:41.339 INFO:teuthology.orchestra.run.smithi042.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T02:13:41.339 INFO:teuthology.orchestra.run.smithi042.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T02:13:41.339 INFO:teuthology.orchestra.run.smithi042.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T02:13:41.339 INFO:teuthology.orchestra.run.smithi042.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T02:13:41.339 INFO:teuthology.orchestra.run.smithi042.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T02:13:41.339 INFO:teuthology.orchestra.run.smithi042.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T02:13:41.339 INFO:teuthology.orchestra.run.smithi042.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T02:13:41.339 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:41.339 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:13:41.722 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-mgr-rook 2024-03-29T02:13:42.296 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:02:13 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:13:42.456 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:13:42.457 INFO:teuthology.orchestra.run.smithi042.stdout:======================================================================================== 2024-03-29T02:13:42.457 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repository Size 2024-03-29T02:13:42.457 INFO:teuthology.orchestra.run.smithi042.stdout:======================================================================================== 2024-03-29T02:13:42.457 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-29T02:13:42.457 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-rook noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 49 k 2024-03-29T02:13:42.457 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-03-29T02:13:42.457 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T02:13:42.457 INFO:teuthology.orchestra.run.smithi042.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T02:13:42.457 INFO:teuthology.orchestra.run.smithi042.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T02:13:42.457 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T02:13:42.458 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T02:13:42.458 INFO:teuthology.orchestra.run.smithi042.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T02:13:42.458 INFO:teuthology.orchestra.run.smithi042.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T02:13:42.458 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T02:13:42.458 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T02:13:42.458 INFO:teuthology.orchestra.run.smithi042.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T02:13:42.458 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T02:13:42.458 INFO:teuthology.orchestra.run.smithi042.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T02:13:42.458 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:42.458 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-29T02:13:42.458 INFO:teuthology.orchestra.run.smithi042.stdout:======================================================================================== 2024-03-29T02:13:42.458 INFO:teuthology.orchestra.run.smithi042.stdout:Install 13 Packages 2024-03-29T02:13:42.458 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:42.459 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 2.0 M 2024-03-29T02:13:42.459 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 26 M 2024-03-29T02:13:42.459 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-29T02:13:42.958 INFO:teuthology.orchestra.run.smithi042.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 173 kB/s | 24 kB 00:00 2024-03-29T02:13:42.983 INFO:teuthology.orchestra.run.smithi042.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 109 kB/s | 18 kB 00:00 2024-03-29T02:13:43.066 INFO:teuthology.orchestra.run.smithi042.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-03-29T02:13:43.092 INFO:teuthology.orchestra.run.smithi042.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 132 kB 00:00 2024-03-29T02:13:43.117 INFO:teuthology.orchestra.run.smithi042.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T02:13:43.134 INFO:teuthology.orchestra.run.smithi042.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-29T02:13:43.151 INFO:teuthology.orchestra.run.smithi042.stdout:(7/13): ceph-mgr-rook-19.0.0-2473.g785fc5e8.el9 147 kB/s | 49 kB 00:00 2024-03-29T02:13:43.193 INFO:teuthology.orchestra.run.smithi042.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 255 kB/s | 15 kB 00:00 2024-03-29T02:13:43.226 INFO:teuthology.orchestra.run.smithi042.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 295 kB/s | 32 kB 00:00 2024-03-29T02:13:43.493 INFO:teuthology.orchestra.run.smithi042.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 222 kB/s | 59 kB 00:00 2024-03-29T02:13:43.544 INFO:teuthology.orchestra.run.smithi042.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 516 kB/s | 203 kB 00:00 2024-03-29T02:13:43.586 INFO:teuthology.orchestra.run.smithi042.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 974 kB/s | 90 kB 00:00 2024-03-29T02:13:43.736 INFO:teuthology.orchestra.run.smithi042.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2024-03-29T02:13:43.738 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:13:43.738 INFO:teuthology.orchestra.run.smithi042.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-29T02:13:43.739 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-29T02:13:43.755 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-29T02:13:43.755 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-29T02:13:43.860 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-29T02:13:43.860 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-29T02:13:44.386 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-29T02:13:44.557 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T02:13:44.613 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T02:13:44.677 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T02:13:44.722 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T02:13:44.764 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T02:13:44.871 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T02:13:44.982 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T02:13:45.033 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T02:13:45.799 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T02:13:45.845 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T02:13:45.879 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T02:13:45.928 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T02:13:45.957 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-rook-2:19.0.0-2473.g785fc5e8.el9.noarch 13/13 2024-03-29T02:13:46.718 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2473.g785fc5e8.el9.noarch 13/13 2024-03-29T02:13:46.719 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2473.g785fc5e8.el9.noarch 1/13 2024-03-29T02:13:46.719 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T02:13:46.719 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T02:13:46.719 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T02:13:46.719 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T02:13:46.719 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T02:13:46.719 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T02:13:46.719 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T02:13:46.719 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T02:13:46.719 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T02:13:46.720 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T02:13:46.720 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T02:13:47.096 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T02:13:47.096 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:47.096 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-29T02:13:47.096 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-rook-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:13:47.096 INFO:teuthology.orchestra.run.smithi042.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T02:13:47.096 INFO:teuthology.orchestra.run.smithi042.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T02:13:47.096 INFO:teuthology.orchestra.run.smithi042.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T02:13:47.096 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T02:13:47.096 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T02:13:47.097 INFO:teuthology.orchestra.run.smithi042.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T02:13:47.097 INFO:teuthology.orchestra.run.smithi042.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T02:13:47.097 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T02:13:47.097 INFO:teuthology.orchestra.run.smithi042.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T02:13:47.097 INFO:teuthology.orchestra.run.smithi042.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T02:13:47.097 INFO:teuthology.orchestra.run.smithi042.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T02:13:47.097 INFO:teuthology.orchestra.run.smithi042.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T02:13:47.097 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:47.097 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:13:47.329 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T02:13:47.907 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:02:18 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:13:48.065 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:13:48.066 INFO:teuthology.orchestra.run.smithi042.stdout:======================================================================================== 2024-03-29T02:13:48.066 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repository Size 2024-03-29T02:13:48.066 INFO:teuthology.orchestra.run.smithi042.stdout:======================================================================================== 2024-03-29T02:13:48.066 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-29T02:13:48.066 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 141 k 2024-03-29T02:13:48.066 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-03-29T02:13:48.066 INFO:teuthology.orchestra.run.smithi042.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T02:13:48.066 INFO:teuthology.orchestra.run.smithi042.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T02:13:48.066 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T02:13:48.066 INFO:teuthology.orchestra.run.smithi042.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T02:13:48.066 INFO:teuthology.orchestra.run.smithi042.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T02:13:48.066 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:48.067 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-29T02:13:48.067 INFO:teuthology.orchestra.run.smithi042.stdout:======================================================================================== 2024-03-29T02:13:48.067 INFO:teuthology.orchestra.run.smithi042.stdout:Install 6 Packages 2024-03-29T02:13:48.067 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:48.067 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 6.8 M 2024-03-29T02:13:48.067 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 33 M 2024-03-29T02:13:48.068 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-29T02:13:48.596 INFO:teuthology.orchestra.run.smithi042.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-03-29T02:13:48.888 INFO:teuthology.orchestra.run.smithi042.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 5.8 MB 00:00 2024-03-29T02:13:48.915 INFO:teuthology.orchestra.run.smithi042.stdout:(3/6): ceph-mgr-cephadm-19.0.0-2473.g785fc5e8.e 268 kB/s | 141 kB 00:00 2024-03-29T02:13:49.149 INFO:teuthology.orchestra.run.smithi042.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 222 kB/s | 58 kB 00:00 2024-03-29T02:13:49.174 INFO:teuthology.orchestra.run.smithi042.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 197 kB/s | 51 kB 00:00 2024-03-29T02:13:49.199 INFO:teuthology.orchestra.run.smithi042.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 910 kB/s | 548 kB 00:00 2024-03-29T02:13:49.203 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:13:49.203 INFO:teuthology.orchestra.run.smithi042.stdout:Total 6.0 MB/s | 6.8 MB 00:01 2024-03-29T02:13:49.216 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-29T02:13:49.226 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-29T02:13:49.226 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-29T02:13:49.318 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-29T02:13:49.319 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-29T02:13:49.622 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-29T02:13:49.806 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T02:13:49.869 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T02:13:50.417 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T02:13:50.507 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T02:13:50.576 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T02:13:50.613 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 6/6 2024-03-29T02:13:51.437 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 6/6 2024-03-29T02:13:51.438 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 1/6 2024-03-29T02:13:51.438 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T02:13:51.438 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T02:13:51.438 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T02:13:51.438 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T02:13:51.734 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T02:13:51.734 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:51.734 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-29T02:13:51.734 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-mgr-cephadm-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:13:51.734 INFO:teuthology.orchestra.run.smithi042.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T02:13:51.734 INFO:teuthology.orchestra.run.smithi042.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T02:13:51.735 INFO:teuthology.orchestra.run.smithi042.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T02:13:51.735 INFO:teuthology.orchestra.run.smithi042.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T02:13:51.735 INFO:teuthology.orchestra.run.smithi042.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T02:13:51.735 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:51.735 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:13:51.993 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-fuse 2024-03-29T02:13:52.568 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:02:23 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:13:52.727 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:13:52.728 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:13:52.728 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repository Size 2024-03-29T02:13:52.728 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:13:52.728 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-29T02:13:52.728 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-fuse x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 871 k 2024-03-29T02:13:52.728 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-03-29T02:13:52.728 INFO:teuthology.orchestra.run.smithi042.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T02:13:52.728 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:52.728 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-29T02:13:52.729 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:13:52.729 INFO:teuthology.orchestra.run.smithi042.stdout:Install 2 Packages 2024-03-29T02:13:52.729 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:52.729 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 949 k 2024-03-29T02:13:52.729 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 4.0 M 2024-03-29T02:13:52.729 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-29T02:13:52.868 INFO:teuthology.orchestra.run.smithi042.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 566 kB/s | 79 kB 00:00 2024-03-29T02:13:53.235 INFO:teuthology.orchestra.run.smithi042.stdout:(2/2): ceph-fuse-19.0.0-2473.g785fc5e8.el9.x86_ 1.7 MB/s | 871 kB 00:00 2024-03-29T02:13:53.236 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:13:53.236 INFO:teuthology.orchestra.run.smithi042.stdout:Total 1.8 MB/s | 949 kB 00:00 2024-03-29T02:13:53.236 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-29T02:13:53.260 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-29T02:13:53.260 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-29T02:13:53.335 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-29T02:13:53.335 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-29T02:13:53.523 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-29T02:13:53.674 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T02:13:53.755 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/2 2024-03-29T02:13:54.420 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/2 2024-03-29T02:13:54.420 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/2 2024-03-29T02:13:54.726 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T02:13:54.726 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:54.726 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-29T02:13:54.726 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T02:13:54.727 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:54.727 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:13:55.037 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install ceph-volume 2024-03-29T02:13:55.614 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:02:26 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:13:55.777 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:13:55.778 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:13:55.778 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repository Size 2024-03-29T02:13:55.778 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:13:55.778 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-29T02:13:55.778 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-volume noarch 2:19.0.0-2473.g785fc5e8.el9 ceph-noarch 264 k 2024-03-29T02:13:55.778 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-03-29T02:13:55.778 INFO:teuthology.orchestra.run.smithi042.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T02:13:55.778 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:55.779 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-29T02:13:55.779 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:13:55.779 INFO:teuthology.orchestra.run.smithi042.stdout:Install 2 Packages 2024-03-29T02:13:55.779 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:55.779 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 332 k 2024-03-29T02:13:55.779 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 1.5 M 2024-03-29T02:13:55.779 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-29T02:13:55.906 INFO:teuthology.orchestra.run.smithi042.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 541 kB/s | 69 kB 00:00 2024-03-29T02:13:56.190 INFO:teuthology.orchestra.run.smithi042.stdout:(2/2): ceph-volume-19.0.0-2473.g785fc5e8.el9.no 642 kB/s | 264 kB 00:00 2024-03-29T02:13:56.191 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:13:56.191 INFO:teuthology.orchestra.run.smithi042.stdout:Total 805 kB/s | 332 kB 00:00 2024-03-29T02:13:56.191 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-29T02:13:56.201 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-29T02:13:56.201 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-29T02:13:56.246 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-29T02:13:56.246 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-29T02:13:56.471 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-29T02:13:56.645 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T02:13:56.693 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : ceph-volume-2:19.0.0-2473.g785fc5e8.el9.noarch 2/2 2024-03-29T02:13:56.708 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: ceph-volume-2:19.0.0-2473.g785fc5e8.el9.noarch 2/2 2024-03-29T02:13:56.709 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T02:13:56.709 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T02:13:56.709 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:57.526 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : ceph-volume-2:19.0.0-2473.g785fc5e8.el9.noarch 1/2 2024-03-29T02:13:57.859 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T02:13:57.859 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:57.859 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-29T02:13:57.859 INFO:teuthology.orchestra.run.smithi042.stdout: ceph-volume-2:19.0.0-2473.g785fc5e8.el9.noarch 2024-03-29T02:13:57.860 INFO:teuthology.orchestra.run.smithi042.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T02:13:57.860 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:57.860 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:13:58.167 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install librados-devel 2024-03-29T02:13:58.751 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:02:29 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:13:58.909 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:13:58.910 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:13:58.910 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-29T02:13:58.910 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:13:58.910 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-29T02:13:58.910 INFO:teuthology.orchestra.run.smithi042.stdout: librados-devel x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 125 k 2024-03-29T02:13:58.910 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:58.910 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-29T02:13:58.910 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:13:58.910 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-29T02:13:58.910 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:13:58.910 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 125 k 2024-03-29T02:13:58.910 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 445 k 2024-03-29T02:13:58.910 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-29T02:13:59.296 INFO:teuthology.orchestra.run.smithi042.stdout:librados-devel-19.0.0-2473.g785fc5e8.el9.x86_64 324 kB/s | 125 kB 00:00 2024-03-29T02:13:59.296 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:13:59.296 INFO:teuthology.orchestra.run.smithi042.stdout:Total 322 kB/s | 125 kB 00:00 2024-03-29T02:13:59.296 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-29T02:13:59.303 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-29T02:13:59.303 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-29T02:13:59.362 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-29T02:13:59.362 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-29T02:13:59.522 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-29T02:13:59.622 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : librados-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:14:00.089 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: librados-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:14:00.382 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librados-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:14:00.382 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:00.382 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-29T02:14:00.382 INFO:teuthology.orchestra.run.smithi042.stdout: librados-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:14:00.382 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:00.382 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:14:00.599 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install libcephfs2 2024-03-29T02:14:01.179 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:02:32 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:14:01.282 INFO:teuthology.orchestra.run.smithi042.stdout:Package libcephfs2-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:14:01.338 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:14:01.339 INFO:teuthology.orchestra.run.smithi042.stdout:Nothing to do. 2024-03-29T02:14:01.339 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:14:01.412 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install libcephfs-devel 2024-03-29T02:14:01.986 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:02:32 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:14:02.143 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:14:02.144 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:14:02.144 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-29T02:14:02.144 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:14:02.144 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-29T02:14:02.144 INFO:teuthology.orchestra.run.smithi042.stdout: libcephfs-devel x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 32 k 2024-03-29T02:14:02.144 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:02.144 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-29T02:14:02.145 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:14:02.145 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-29T02:14:02.145 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:02.145 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 32 k 2024-03-29T02:14:02.145 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 142 k 2024-03-29T02:14:02.145 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-29T02:14:02.599 INFO:teuthology.orchestra.run.smithi042.stdout:libcephfs-devel-19.0.0-2473.g785fc5e8.el9.x86_6 70 kB/s | 32 kB 00:00 2024-03-29T02:14:02.599 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:14:02.600 INFO:teuthology.orchestra.run.smithi042.stdout:Total 70 kB/s | 32 kB 00:00 2024-03-29T02:14:02.600 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-29T02:14:02.603 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-29T02:14:02.603 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-29T02:14:02.633 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-29T02:14:02.633 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-29T02:14:02.719 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-29T02:14:02.841 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libcephfs-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:14:03.240 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:14:03.497 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libcephfs-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:14:03.497 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:03.497 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-29T02:14:03.497 INFO:teuthology.orchestra.run.smithi042.stdout: libcephfs-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:14:03.497 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:03.497 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:14:03.674 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install librados2 2024-03-29T02:14:04.249 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:02:35 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:14:04.350 INFO:teuthology.orchestra.run.smithi042.stdout:Package librados2-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:14:04.406 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:14:04.406 INFO:teuthology.orchestra.run.smithi042.stdout:Nothing to do. 2024-03-29T02:14:04.406 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:14:04.473 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install librbd1 2024-03-29T02:14:05.046 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:02:36 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:14:05.147 INFO:teuthology.orchestra.run.smithi042.stdout:Package librbd1-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:14:05.204 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:14:05.204 INFO:teuthology.orchestra.run.smithi042.stdout:Nothing to do. 2024-03-29T02:14:05.204 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:14:05.278 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install python3-rados 2024-03-29T02:14:05.853 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:02:36 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:14:05.955 INFO:teuthology.orchestra.run.smithi042.stdout:Package python3-rados-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:14:06.010 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:14:06.012 INFO:teuthology.orchestra.run.smithi042.stdout:Nothing to do. 2024-03-29T02:14:06.012 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:14:06.083 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install python3-rgw 2024-03-29T02:14:06.657 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:02:37 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:14:06.758 INFO:teuthology.orchestra.run.smithi042.stdout:Package python3-rgw-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:14:06.814 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:14:06.815 INFO:teuthology.orchestra.run.smithi042.stdout:Nothing to do. 2024-03-29T02:14:06.815 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:14:06.886 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install python3-cephfs 2024-03-29T02:14:07.466 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:02:38 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:14:07.569 INFO:teuthology.orchestra.run.smithi042.stdout:Package python3-cephfs-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:14:07.624 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:14:07.625 INFO:teuthology.orchestra.run.smithi042.stdout:Nothing to do. 2024-03-29T02:14:07.625 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:14:07.699 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install python3-rbd 2024-03-29T02:14:08.278 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:02:39 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:14:08.381 INFO:teuthology.orchestra.run.smithi042.stdout:Package python3-rbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 is already installed. 2024-03-29T02:14:08.437 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:14:08.438 INFO:teuthology.orchestra.run.smithi042.stdout:Nothing to do. 2024-03-29T02:14:08.438 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:14:08.513 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install rbd-fuse 2024-03-29T02:14:09.095 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:02:40 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:14:09.252 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:14:09.253 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:14:09.253 INFO:teuthology.orchestra.run.smithi042.stdout: Package Architecture Version Repository Size 2024-03-29T02:14:09.253 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:14:09.253 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-29T02:14:09.253 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-fuse x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 90 k 2024-03-29T02:14:09.253 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:09.253 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-29T02:14:09.253 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:14:09.253 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-29T02:14:09.254 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:09.254 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 90 k 2024-03-29T02:14:09.254 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 313 k 2024-03-29T02:14:09.254 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-29T02:14:09.469 INFO:teuthology.orchestra.run.smithi042.stdout:rbd-fuse-19.0.0-2473.g785fc5e8.el9.x86_64.rpm 417 kB/s | 90 kB 00:00 2024-03-29T02:14:09.469 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:14:09.470 INFO:teuthology.orchestra.run.smithi042.stdout:Total 414 kB/s | 90 kB 00:00 2024-03-29T02:14:09.470 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-29T02:14:09.480 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-29T02:14:09.480 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-29T02:14:09.537 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-29T02:14:09.537 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-29T02:14:09.683 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-29T02:14:09.795 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : rbd-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:14:10.195 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:14:10.498 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : rbd-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:14:10.498 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:10.498 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-29T02:14:10.498 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-fuse-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:14:10.499 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:10.499 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:14:10.748 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install rbd-mirror 2024-03-29T02:14:11.327 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:02:42 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:14:11.484 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:14:11.485 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:14:11.485 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repo Size 2024-03-29T02:14:11.485 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:14:11.485 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-29T02:14:11.485 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-mirror x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 3.4 M 2024-03-29T02:14:11.486 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:11.486 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-29T02:14:11.486 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:14:11.486 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-29T02:14:11.486 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:11.486 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 3.4 M 2024-03-29T02:14:11.486 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 29 M 2024-03-29T02:14:11.486 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-29T02:14:12.939 INFO:teuthology.orchestra.run.smithi042.stdout:rbd-mirror-19.0.0-2473.g785fc5e8.el9.x86_64.rpm 2.3 MB/s | 3.4 MB 00:01 2024-03-29T02:14:12.940 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:14:12.940 INFO:teuthology.orchestra.run.smithi042.stdout:Total 2.3 MB/s | 3.4 MB 00:01 2024-03-29T02:14:12.940 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-29T02:14:12.954 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-29T02:14:12.954 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-29T02:14:13.036 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-29T02:14:13.037 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-29T02:14:13.633 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-29T02:14:13.681 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : rbd-mirror-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:14:13.705 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:14:13.706 INFO:teuthology.orchestra.run.smithi042.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T02:14:13.706 INFO:teuthology.orchestra.run.smithi042.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T02:14:13.706 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T02:14:13.706 INFO:teuthology.orchestra.run.smithi042.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T02:14:13.706 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:14.936 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : rbd-mirror-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:14:14.936 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:14.937 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-29T02:14:14.937 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-mirror-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:14:14.937 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:14.937 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:14:15.194 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install rbd-nbd 2024-03-29T02:14:15.773 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:02:46 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:14:15.932 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:14:15.932 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:14:15.932 INFO:teuthology.orchestra.run.smithi042.stdout: Package Architecture Version Repository Size 2024-03-29T02:14:15.932 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:14:15.932 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-29T02:14:15.932 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-nbd x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 176 k 2024-03-29T02:14:15.932 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:15.932 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-29T02:14:15.932 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:14:15.932 INFO:teuthology.orchestra.run.smithi042.stdout:Install 1 Package 2024-03-29T02:14:15.932 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:15.933 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 176 k 2024-03-29T02:14:15.933 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 673 k 2024-03-29T02:14:15.933 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-29T02:14:16.151 INFO:teuthology.orchestra.run.smithi042.stdout:rbd-nbd-19.0.0-2473.g785fc5e8.el9.x86_64.rpm 808 kB/s | 176 kB 00:00 2024-03-29T02:14:16.151 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:14:16.151 INFO:teuthology.orchestra.run.smithi042.stdout:Total 803 kB/s | 176 kB 00:00 2024-03-29T02:14:16.151 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-29T02:14:16.162 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-29T02:14:16.162 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-29T02:14:16.221 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-29T02:14:16.221 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-29T02:14:16.376 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-29T02:14:16.487 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : rbd-nbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:14:16.926 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:14:17.214 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : rbd-nbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/1 2024-03-29T02:14:17.214 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:17.214 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-29T02:14:17.214 INFO:teuthology.orchestra.run.smithi042.stdout: rbd-nbd-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:14:17.214 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:17.215 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:14:17.472 DEBUG:teuthology.parallel:result is None 2024-03-29T02:14:17.473 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=785fc5e8d093374f751c7da3e51e9b56b3718d17 2024-03-29T02:14:17.616 DEBUG:teuthology.orchestra.run.smithi042:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T02:14:17.637 INFO:teuthology.orchestra.run.smithi042.stdout:19.0.0-2473.g785fc5e8.el9 2024-03-29T02:14:17.637 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2473.g785fc5e8.el9 2024-03-29T02:14:17.638 INFO:teuthology.task.install:The correct ceph version 19.0.0-2473.g785fc5e8 is installed. 2024-03-29T02:14:17.640 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T02:14:17.640 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:14:17.640 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T02:14:17.704 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T02:14:17.705 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:14:17.705 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T02:14:17.771 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T02:14:17.837 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T02:14:17.838 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:14:17.838 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T02:14:17.904 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T02:14:17.970 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T02:14:17.971 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:14:17.971 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T02:14:18.037 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T02:14:18.103 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T02:14:18.192 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T02:14:18.192 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 777 /var/log/ceph 2024-03-29T02:14:18.237 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T02:14:18.237 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T02:14:18.263 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T02:14:18.263 DEBUG:teuthology.orchestra.run.smithi042:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T02:14:18.337 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T02:14:18.337 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-29T02:14:18.338 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629056', 'branch': 'ci-yingxin-crimson-load-obc-3', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_rand_read}', 'email': 'yingxin.cheng@intel.com', 'first_in_suite': False, 'job_id': '7629056', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'ci-yingxin-crimson-load-obc-3'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'ci-yingxin-crimson-load-obc-3', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17'}}, 'owner': 'scheduled_yingxin@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'ci-yingxin-crimson-load-obc-3', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_785fc5e8d093374f751c7da3e51e9b56b3718d17/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17', 'targets': {'smithi042.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFCz6C5cOWFTqJqpj0ql4JPdLVDLqBcXUMM+G+y8b/mr3qrh/KxRgnL5ln2HBa98TihERnoEY1x4X2AZXE+15ZY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '785fc5e8d093374f751c7da3e51e9b56b3718d17', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-29_01:54:41', 'tube': 'smithi', 'user': 'yingxin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T02:14:18.338 DEBUG:teuthology.orchestra.run.smithi042:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T02:14:18.384 DEBUG:teuthology.orchestra.run.smithi042:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T02:14:18.451 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:14:18.451 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T02:14:18.505 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T02:14:18.505 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_1 2024-03-29T02:14:18.561 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T02:14:18.561 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T02:14:18.561 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T02:14:18.561 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T02:14:18.561 INFO:teuthology.orchestra.run.smithi042.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T02:14:18.561 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-03-29 02:14:13.944984464 +0000 2024-03-29T02:14:18.561 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-03-29 02:10:21.354536885 +0000 2024-03-29T02:14:18.561 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-03-29 02:10:21.354536885 +0000 2024-03-29T02:14:18.561 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: 2024-03-29 02:10:21.354536885 +0000 2024-03-29T02:14:18.562 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T02:14:18.625 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2024-03-29T02:14:18.626 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2024-03-29T02:14:18.626 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 9.3181e-05 s, 5.5 MB/s 2024-03-29T02:14:18.627 DEBUG:teuthology.orchestra.run.smithi042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T02:14:18.682 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_2 2024-03-29T02:14:18.736 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T02:14:18.736 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T02:14:18.737 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T02:14:18.737 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T02:14:18.737 INFO:teuthology.orchestra.run.smithi042.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T02:14:18.737 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-03-29 02:14:13.944984464 +0000 2024-03-29T02:14:18.737 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-03-29 02:10:21.677534996 +0000 2024-03-29T02:14:18.737 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-03-29 02:10:21.677534996 +0000 2024-03-29T02:14:18.737 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: 2024-03-29 02:10:21.677534996 +0000 2024-03-29T02:14:18.737 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T02:14:18.802 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2024-03-29T02:14:18.802 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2024-03-29T02:14:18.802 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 0.000115488 s, 4.4 MB/s 2024-03-29T02:14:18.803 DEBUG:teuthology.orchestra.run.smithi042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T02:14:18.859 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_3 2024-03-29T02:14:18.913 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T02:14:18.914 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T02:14:18.914 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-29T02:14:18.914 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T02:14:18.914 INFO:teuthology.orchestra.run.smithi042.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T02:14:18.914 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-03-29 02:14:13.944984464 +0000 2024-03-29T02:14:18.914 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-03-29 02:10:22.021532985 +0000 2024-03-29T02:14:18.914 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-03-29 02:10:22.021532985 +0000 2024-03-29T02:14:18.914 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: 2024-03-29 02:10:22.021532985 +0000 2024-03-29T02:14:18.914 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T02:14:18.980 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2024-03-29T02:14:18.980 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2024-03-29T02:14:18.980 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 7.9691e-05 s, 6.4 MB/s 2024-03-29T02:14:18.980 DEBUG:teuthology.orchestra.run.smithi042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T02:14:19.035 DEBUG:teuthology.orchestra.run.smithi042:> stat /dev/vg_nvme/lv_4 2024-03-29T02:14:19.089 INFO:teuthology.orchestra.run.smithi042.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T02:14:19.089 INFO:teuthology.orchestra.run.smithi042.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T02:14:19.089 INFO:teuthology.orchestra.run.smithi042.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-29T02:14:19.089 INFO:teuthology.orchestra.run.smithi042.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T02:14:19.089 INFO:teuthology.orchestra.run.smithi042.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T02:14:19.089 INFO:teuthology.orchestra.run.smithi042.stdout:Access: 2024-03-29 02:14:13.945984457 +0000 2024-03-29T02:14:19.089 INFO:teuthology.orchestra.run.smithi042.stdout:Modify: 2024-03-29 02:10:22.359531009 +0000 2024-03-29T02:14:19.089 INFO:teuthology.orchestra.run.smithi042.stdout:Change: 2024-03-29 02:10:22.359531009 +0000 2024-03-29T02:14:19.090 INFO:teuthology.orchestra.run.smithi042.stdout: Birth: 2024-03-29 02:10:22.359531009 +0000 2024-03-29T02:14:19.090 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T02:14:19.153 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records in 2024-03-29T02:14:19.153 INFO:teuthology.orchestra.run.smithi042.stderr:1+0 records out 2024-03-29T02:14:19.153 INFO:teuthology.orchestra.run.smithi042.stderr:512 bytes copied, 9.3643e-05 s, 5.5 MB/s 2024-03-29T02:14:19.155 DEBUG:teuthology.orchestra.run.smithi042:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T02:14:19.210 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-03-29T02:14:19.210 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi042.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-29T02:14:19.210 INFO:tasks.ceph:Generating config... 2024-03-29T02:14:19.211 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T02:14:19.211 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T02:14:19.211 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-29T02:14:19.212 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T02:14:19.212 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T02:14:19.212 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-03-29T02:14:19.212 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-03-29T02:14:19.212 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-03-29T02:14:19.212 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T02:14:19.212 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T02:14:19.212 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-03-29T02:14:19.212 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-03-29T02:14:19.212 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-03-29T02:14:19.212 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-03-29T02:14:19.212 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T02:14:19.212 INFO:tasks.ceph:[osd] err to stderr = True 2024-03-29T02:14:19.212 INFO:tasks.ceph:[osd] log flush on exit = True 2024-03-29T02:14:19.213 INFO:tasks.ceph:[osd] log to file = False 2024-03-29T02:14:19.213 INFO:tasks.ceph:[osd] log to stderr = True 2024-03-29T02:14:19.213 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T02:14:19.213 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T02:14:19.213 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T02:14:19.213 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T02:14:19.213 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T02:14:19.213 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T02:14:19.213 INFO:tasks.ceph:[global] auth client required = none 2024-03-29T02:14:19.213 INFO:tasks.ceph:[global] auth cluster required = none 2024-03-29T02:14:19.213 INFO:tasks.ceph:[global] auth service required = none 2024-03-29T02:14:19.213 INFO:tasks.ceph:[global] auth supported = none 2024-03-29T02:14:19.213 INFO:tasks.ceph:[global] debug asok = 0/0 2024-03-29T02:14:19.213 INFO:tasks.ceph:[global] debug auth = 0/0 2024-03-29T02:14:19.214 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-03-29T02:14:19.214 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-03-29T02:14:19.214 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-03-29T02:14:19.214 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-03-29T02:14:19.214 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-03-29T02:14:19.214 INFO:tasks.ceph:[global] debug client = 0/0 2024-03-29T02:14:19.214 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-03-29T02:14:19.214 INFO:tasks.ceph:[global] debug context = 0/0 2024-03-29T02:14:19.214 INFO:tasks.ceph:[global] debug crush = 0/0 2024-03-29T02:14:19.214 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-03-29T02:14:19.214 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-03-29T02:14:19.214 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-03-29T02:14:19.214 INFO:tasks.ceph:[global] debug filer = 0/0 2024-03-29T02:14:19.214 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-03-29T02:14:19.215 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-03-29T02:14:19.215 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-03-29T02:14:19.215 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-03-29T02:14:19.215 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-03-29T02:14:19.215 INFO:tasks.ceph:[global] debug journal = 0/0 2024-03-29T02:14:19.215 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-03-29T02:14:19.215 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-03-29T02:14:19.215 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-03-29T02:14:19.215 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-03-29T02:14:19.215 INFO:tasks.ceph:[global] debug mds = 0/0 2024-03-29T02:14:19.215 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-03-29T02:14:19.215 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-03-29T02:14:19.215 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-03-29T02:14:19.215 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-03-29T02:14:19.215 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-03-29T02:14:19.216 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-03-29T02:14:19.216 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-03-29T02:14:19.216 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-03-29T02:14:19.216 INFO:tasks.ceph:[global] debug mon = 0/0 2024-03-29T02:14:19.216 INFO:tasks.ceph:[global] debug monc = 0/0 2024-03-29T02:14:19.216 INFO:tasks.ceph:[global] debug ms = 0/0 2024-03-29T02:14:19.216 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-03-29T02:14:19.216 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-03-29T02:14:19.216 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-03-29T02:14:19.216 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-03-29T02:14:19.216 INFO:tasks.ceph:[global] debug osd = 0/0 2024-03-29T02:14:19.216 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-03-29T02:14:19.216 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-03-29T02:14:19.216 INFO:tasks.ceph:[global] debug rados = 0/0 2024-03-29T02:14:19.216 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-03-29T02:14:19.217 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-03-29T02:14:19.217 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-03-29T02:14:19.217 INFO:tasks.ceph:[global] debug refs = 0/0 2024-03-29T02:14:19.217 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-03-29T02:14:19.217 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2024-03-29T02:14:19.217 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2024-03-29T02:14:19.217 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-03-29T02:14:19.217 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-03-29T02:14:19.217 INFO:tasks.ceph:[global] debug striper = 0/0 2024-03-29T02:14:19.217 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-03-29T02:14:19.217 INFO:tasks.ceph:[global] debug timer = 0/0 2024-03-29T02:14:19.217 INFO:tasks.ceph:[global] debug tp = 0/0 2024-03-29T02:14:19.217 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-03-29T02:14:19.217 INFO:tasks.ceph:[global] ms client mode = crc 2024-03-29T02:14:19.218 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-03-29T02:14:19.218 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-03-29T02:14:19.218 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-03-29T02:14:19.218 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-03-29T02:14:19.218 INFO:tasks.ceph:[global] ms service mode = crc 2024-03-29T02:14:19.218 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T02:14:19.218 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T02:14:19.218 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-03-29T02:14:19.218 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-03-29T02:14:19.218 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-03-29T02:14:19.218 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-03-29T02:14:19.218 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-03-29T02:14:19.218 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-03-29T02:14:19.218 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-03-29T02:14:19.219 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-03-29T02:14:19.219 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T02:14:19.219 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T02:14:19.313 INFO:teuthology.orchestra.run.smithi042.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T02:14:19.316 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T02:14:19.370 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T02:14:19.396 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.42')] 2024-03-29T02:14:19.396 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.42', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-03-29T02:14:19.397 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:14:19.398 DEBUG:teuthology.orchestra.run.smithi042:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T02:14:19.450 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.42 --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T02:14:19.542 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:19.542+0000 7f6a1688ff00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:19.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:19.542+0000 7f6a1688ff00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:19.543 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:19.542+0000 7f6a1688ff00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:19.543 INFO:teuthology.orchestra.run.smithi042.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T02:14:19.543 INFO:teuthology.orchestra.run.smithi042.stdout:monmaptool: generated fsid 4ad80ad0-2c97-4ca8-9f08-3f0610eec781 2024-03-29T02:14:19.544 INFO:teuthology.orchestra.run.smithi042.stdout:setting min_mon_release = quincy 2024-03-29T02:14:19.544 INFO:teuthology.orchestra.run.smithi042.stdout:epoch 0 2024-03-29T02:14:19.544 INFO:teuthology.orchestra.run.smithi042.stdout:fsid 4ad80ad0-2c97-4ca8-9f08-3f0610eec781 2024-03-29T02:14:19.544 INFO:teuthology.orchestra.run.smithi042.stdout:last_changed 2024-03-29T02:14:19.544708+0000 2024-03-29T02:14:19.544 INFO:teuthology.orchestra.run.smithi042.stdout:created 2024-03-29T02:14:19.544708+0000 2024-03-29T02:14:19.544 INFO:teuthology.orchestra.run.smithi042.stdout:min_mon_release 17 (quincy) 2024-03-29T02:14:19.544 INFO:teuthology.orchestra.run.smithi042.stdout:election_strategy: 1 2024-03-29T02:14:19.544 INFO:teuthology.orchestra.run.smithi042.stdout:0: [v2:172.21.15.42:3300/0,v1:172.21.15.42:6789/0] mon.a 2024-03-29T02:14:19.544 INFO:teuthology.orchestra.run.smithi042.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-03-29T02:14:19.545 DEBUG:teuthology.orchestra.run.smithi042:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T02:14:19.559 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 4ad80ad0-2c97-4ca8-9f08-3f0610eec781... 2024-03-29T02:14:19.560 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T02:14:19.650 INFO:teuthology.orchestra.run.smithi042.stdout:[global] 2024-03-29T02:14:19.650 INFO:teuthology.orchestra.run.smithi042.stdout: chdir = "" 2024-03-29T02:14:19.650 INFO:teuthology.orchestra.run.smithi042.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T02:14:19.651 INFO:teuthology.orchestra.run.smithi042.stdout: auth supported = none 2024-03-29T02:14:19.651 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.651 INFO:teuthology.orchestra.run.smithi042.stdout: filestore xattr use omap = true 2024-03-29T02:14:19.651 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.651 INFO:teuthology.orchestra.run.smithi042.stdout: mon clock drift allowed = 1.000 2024-03-29T02:14:19.651 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.651 INFO:teuthology.orchestra.run.smithi042.stdout: osd crush chooseleaf type = 0 2024-03-29T02:14:19.651 INFO:teuthology.orchestra.run.smithi042.stdout: auth debug = true 2024-03-29T02:14:19.651 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.652 INFO:teuthology.orchestra.run.smithi042.stdout: ms die on old message = true 2024-03-29T02:14:19.652 INFO:teuthology.orchestra.run.smithi042.stdout: ms die on bug = true 2024-03-29T02:14:19.652 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.652 INFO:teuthology.orchestra.run.smithi042.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T02:14:19.652 INFO:teuthology.orchestra.run.smithi042.stdout: mon pg warn max object skew = 0 2024-03-29T02:14:19.652 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.652 INFO:teuthology.orchestra.run.smithi042.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T02:14:19.652 INFO:teuthology.orchestra.run.smithi042.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T02:14:19.652 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.652 INFO:teuthology.orchestra.run.smithi042.stdout: osd pool default size = 2 2024-03-29T02:14:19.652 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.652 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd allow primary affinity = true 2024-03-29T02:14:19.652 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd allow pg remap = true 2024-03-29T02:14:19.652 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on legacy crush tunables = false 2024-03-29T02:14:19.653 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on crush straw calc version zero = false 2024-03-29T02:14:19.653 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on no sortbitwise = false 2024-03-29T02:14:19.653 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on osd down out interval zero = false 2024-03-29T02:14:19.653 INFO:teuthology.orchestra.run.smithi042.stdout: mon warn on too few osds = false 2024-03-29T02:14:19.653 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T02:14:19.653 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T02:14:19.653 INFO:teuthology.orchestra.run.smithi042.stdout: mon_allow_pool_size_one = true 2024-03-29T02:14:19.653 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.653 INFO:teuthology.orchestra.run.smithi042.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T02:14:19.653 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.653 INFO:teuthology.orchestra.run.smithi042.stdout: osd default data pool replay window = 5 2024-03-29T02:14:19.653 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.653 INFO:teuthology.orchestra.run.smithi042.stdout: mon allow pool delete = true 2024-03-29T02:14:19.653 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.653 INFO:teuthology.orchestra.run.smithi042.stdout: mon cluster log file level = debug 2024-03-29T02:14:19.654 INFO:teuthology.orchestra.run.smithi042.stdout: debug asserts on shutdown = true 2024-03-29T02:14:19.654 INFO:teuthology.orchestra.run.smithi042.stdout: mon health detail to clog = false 2024-03-29T02:14:19.654 INFO:teuthology.orchestra.run.smithi042.stdout: mon host = 172.21.15.42 2024-03-29T02:14:19.654 INFO:teuthology.orchestra.run.smithi042.stdout: auth client required = none 2024-03-29T02:14:19.654 INFO:teuthology.orchestra.run.smithi042.stdout: auth cluster required = none 2024-03-29T02:14:19.654 INFO:teuthology.orchestra.run.smithi042.stdout: auth service required = none 2024-03-29T02:14:19.654 INFO:teuthology.orchestra.run.smithi042.stdout: debug asok = 0/0 2024-03-29T02:14:19.654 INFO:teuthology.orchestra.run.smithi042.stdout: debug auth = 0/0 2024-03-29T02:14:19.654 INFO:teuthology.orchestra.run.smithi042.stdout: debug bdev = 0/0 2024-03-29T02:14:19.654 INFO:teuthology.orchestra.run.smithi042.stdout: debug bluefs = 0/0 2024-03-29T02:14:19.654 INFO:teuthology.orchestra.run.smithi042.stdout: debug bluestore = 0/0 2024-03-29T02:14:19.654 INFO:teuthology.orchestra.run.smithi042.stdout: debug buffer = 0/0 2024-03-29T02:14:19.655 INFO:teuthology.orchestra.run.smithi042.stdout: debug civetweb = 0/0 2024-03-29T02:14:19.655 INFO:teuthology.orchestra.run.smithi042.stdout: debug client = 0/0 2024-03-29T02:14:19.655 INFO:teuthology.orchestra.run.smithi042.stdout: debug compressor = 0/0 2024-03-29T02:14:19.655 INFO:teuthology.orchestra.run.smithi042.stdout: debug context = 0/0 2024-03-29T02:14:19.655 INFO:teuthology.orchestra.run.smithi042.stdout: debug crush = 0/0 2024-03-29T02:14:19.655 INFO:teuthology.orchestra.run.smithi042.stdout: debug crypto = 0/0 2024-03-29T02:14:19.655 INFO:teuthology.orchestra.run.smithi042.stdout: debug dpdk = 0/0 2024-03-29T02:14:19.655 INFO:teuthology.orchestra.run.smithi042.stdout: debug eventtrace = 0/0 2024-03-29T02:14:19.655 INFO:teuthology.orchestra.run.smithi042.stdout: debug filer = 0/0 2024-03-29T02:14:19.655 INFO:teuthology.orchestra.run.smithi042.stdout: debug filestore = 0/0 2024-03-29T02:14:19.655 INFO:teuthology.orchestra.run.smithi042.stdout: debug finisher = 0/0 2024-03-29T02:14:19.655 INFO:teuthology.orchestra.run.smithi042.stdout: debug fuse = 0/0 2024-03-29T02:14:19.655 INFO:teuthology.orchestra.run.smithi042.stdout: debug heartbeatmap = 0/0 2024-03-29T02:14:19.655 INFO:teuthology.orchestra.run.smithi042.stdout: debug javaclient = 0/0 2024-03-29T02:14:19.655 INFO:teuthology.orchestra.run.smithi042.stdout: debug journal = 0/0 2024-03-29T02:14:19.656 INFO:teuthology.orchestra.run.smithi042.stdout: debug journaler = 0/0 2024-03-29T02:14:19.656 INFO:teuthology.orchestra.run.smithi042.stdout: debug kstore = 0/0 2024-03-29T02:14:19.656 INFO:teuthology.orchestra.run.smithi042.stdout: debug leveldb = 0/0 2024-03-29T02:14:19.656 INFO:teuthology.orchestra.run.smithi042.stdout: debug lockdep = 0/0 2024-03-29T02:14:19.656 INFO:teuthology.orchestra.run.smithi042.stdout: debug mds = 0/0 2024-03-29T02:14:19.656 INFO:teuthology.orchestra.run.smithi042.stdout: debug mds balancer = 0/0 2024-03-29T02:14:19.656 INFO:teuthology.orchestra.run.smithi042.stdout: debug mds locker = 0/0 2024-03-29T02:14:19.656 INFO:teuthology.orchestra.run.smithi042.stdout: debug mds log = 0/0 2024-03-29T02:14:19.656 INFO:teuthology.orchestra.run.smithi042.stdout: debug mds log expire = 0/0 2024-03-29T02:14:19.656 INFO:teuthology.orchestra.run.smithi042.stdout: debug mds migrator = 0/0 2024-03-29T02:14:19.656 INFO:teuthology.orchestra.run.smithi042.stdout: debug memdb = 0/0 2024-03-29T02:14:19.657 INFO:teuthology.orchestra.run.smithi042.stdout: debug mgr = 0/0 2024-03-29T02:14:19.657 INFO:teuthology.orchestra.run.smithi042.stdout: debug mgrc = 0/0 2024-03-29T02:14:19.657 INFO:teuthology.orchestra.run.smithi042.stdout: debug mon = 0/0 2024-03-29T02:14:19.657 INFO:teuthology.orchestra.run.smithi042.stdout: debug monc = 0/0 2024-03-29T02:14:19.657 INFO:teuthology.orchestra.run.smithi042.stdout: debug ms = 0/0 2024-03-29T02:14:19.657 INFO:teuthology.orchestra.run.smithi042.stdout: debug objclass = 0/0 2024-03-29T02:14:19.657 INFO:teuthology.orchestra.run.smithi042.stdout: debug objectcacher = 0/0 2024-03-29T02:14:19.657 INFO:teuthology.orchestra.run.smithi042.stdout: debug objecter = 0/0 2024-03-29T02:14:19.657 INFO:teuthology.orchestra.run.smithi042.stdout: debug optracker = 0/0 2024-03-29T02:14:19.657 INFO:teuthology.orchestra.run.smithi042.stdout: debug osd = 0/0 2024-03-29T02:14:19.657 INFO:teuthology.orchestra.run.smithi042.stdout: debug paxos = 0/0 2024-03-29T02:14:19.657 INFO:teuthology.orchestra.run.smithi042.stdout: debug perfcounter = 0/0 2024-03-29T02:14:19.657 INFO:teuthology.orchestra.run.smithi042.stdout: debug rados = 0/0 2024-03-29T02:14:19.657 INFO:teuthology.orchestra.run.smithi042.stdout: debug rbd = 0/0 2024-03-29T02:14:19.657 INFO:teuthology.orchestra.run.smithi042.stdout: debug rbd mirror = 0/0 2024-03-29T02:14:19.658 INFO:teuthology.orchestra.run.smithi042.stdout: debug rbd replay = 0/0 2024-03-29T02:14:19.658 INFO:teuthology.orchestra.run.smithi042.stdout: debug refs = 0/0 2024-03-29T02:14:19.658 INFO:teuthology.orchestra.run.smithi042.stdout: debug rgw = 0/0 2024-03-29T02:14:19.658 INFO:teuthology.orchestra.run.smithi042.stdout: debug rgw lifecycle = 0/0 2024-03-29T02:14:19.658 INFO:teuthology.orchestra.run.smithi042.stdout: debug rgw notification = 0/0 2024-03-29T02:14:19.658 INFO:teuthology.orchestra.run.smithi042.stdout: debug rgw sync = 0/0 2024-03-29T02:14:19.658 INFO:teuthology.orchestra.run.smithi042.stdout: debug rocksdb = 0/0 2024-03-29T02:14:19.658 INFO:teuthology.orchestra.run.smithi042.stdout: debug striper = 0/0 2024-03-29T02:14:19.658 INFO:teuthology.orchestra.run.smithi042.stdout: debug throttle = 0/0 2024-03-29T02:14:19.658 INFO:teuthology.orchestra.run.smithi042.stdout: debug timer = 0/0 2024-03-29T02:14:19.658 INFO:teuthology.orchestra.run.smithi042.stdout: debug tp = 0/0 2024-03-29T02:14:19.658 INFO:teuthology.orchestra.run.smithi042.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-03-29T02:14:19.658 INFO:teuthology.orchestra.run.smithi042.stdout: ms client mode = crc 2024-03-29T02:14:19.658 INFO:teuthology.orchestra.run.smithi042.stdout: ms cluster mode = crc 2024-03-29T02:14:19.659 INFO:teuthology.orchestra.run.smithi042.stdout: ms mon client mode = crc 2024-03-29T02:14:19.659 INFO:teuthology.orchestra.run.smithi042.stdout: ms mon cluster mode = crc 2024-03-29T02:14:19.659 INFO:teuthology.orchestra.run.smithi042.stdout: ms mon service mode = crc 2024-03-29T02:14:19.659 INFO:teuthology.orchestra.run.smithi042.stdout: ms service mode = crc 2024-03-29T02:14:19.659 INFO:teuthology.orchestra.run.smithi042.stdout: fsid = 4ad80ad0-2c97-4ca8-9f08-3f0610eec781 2024-03-29T02:14:19.659 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.659 INFO:teuthology.orchestra.run.smithi042.stdout:[osd] 2024-03-29T02:14:19.659 INFO:teuthology.orchestra.run.smithi042.stdout: osd journal size = 100 2024-03-29T02:14:19.659 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.659 INFO:teuthology.orchestra.run.smithi042.stdout: osd scrub load threshold = 5.0 2024-03-29T02:14:19.659 INFO:teuthology.orchestra.run.smithi042.stdout: osd scrub max interval = 600 2024-03-29T02:14:19.659 INFO:teuthology.orchestra.run.smithi042.stdout: osd mclock profile = high_recovery_ops 2024-03-29T02:14:19.659 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.659 INFO:teuthology.orchestra.run.smithi042.stdout: osd recover clone overlap = true 2024-03-29T02:14:19.659 INFO:teuthology.orchestra.run.smithi042.stdout: osd recovery max chunk = 1048576 2024-03-29T02:14:19.659 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.660 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug shutdown = true 2024-03-29T02:14:19.660 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug op order = true 2024-03-29T02:14:19.660 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug verify stray on activate = true 2024-03-29T02:14:19.660 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug trim objects = true 2024-03-29T02:14:19.660 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.660 INFO:teuthology.orchestra.run.smithi042.stdout: osd open classes on start = true 2024-03-29T02:14:19.660 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug pg log writeout = true 2024-03-29T02:14:19.660 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.660 INFO:teuthology.orchestra.run.smithi042.stdout: osd deep scrub update digest min age = 30 2024-03-29T02:14:19.660 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.660 INFO:teuthology.orchestra.run.smithi042.stdout: osd map max advance = 10 2024-03-29T02:14:19.660 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.660 INFO:teuthology.orchestra.run.smithi042.stdout: journal zero on create = true 2024-03-29T02:14:19.660 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.661 INFO:teuthology.orchestra.run.smithi042.stdout: filestore ondisk finisher threads = 3 2024-03-29T02:14:19.661 INFO:teuthology.orchestra.run.smithi042.stdout: filestore apply finisher threads = 3 2024-03-29T02:14:19.661 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.661 INFO:teuthology.orchestra.run.smithi042.stdout: bdev debug aio = true 2024-03-29T02:14:19.661 INFO:teuthology.orchestra.run.smithi042.stdout: osd debug misdirected ops = true 2024-03-29T02:14:19.661 INFO:teuthology.orchestra.run.smithi042.stdout: debug monc = 20 2024-03-29T02:14:19.661 INFO:teuthology.orchestra.run.smithi042.stdout: bluestore block size = 96636764160 2024-03-29T02:14:19.661 INFO:teuthology.orchestra.run.smithi042.stdout: bluestore compression algorithm = snappy 2024-03-29T02:14:19.661 INFO:teuthology.orchestra.run.smithi042.stdout: bluestore compression mode = aggressive 2024-03-29T02:14:19.662 INFO:teuthology.orchestra.run.smithi042.stdout: bluestore fsck on mount = True 2024-03-29T02:14:19.662 INFO:teuthology.orchestra.run.smithi042.stdout: bluestore rocksdb cf = False 2024-03-29T02:14:19.662 INFO:teuthology.orchestra.run.smithi042.stdout: crimson alien thread cpu cores = 6-7 2024-03-29T02:14:19.662 INFO:teuthology.orchestra.run.smithi042.stdout: crimson osd obc lru size = 10 2024-03-29T02:14:19.662 INFO:teuthology.orchestra.run.smithi042.stdout: debug bluefs = 20 2024-03-29T02:14:19.662 INFO:teuthology.orchestra.run.smithi042.stdout: debug bluestore = 20 2024-03-29T02:14:19.662 INFO:teuthology.orchestra.run.smithi042.stdout: debug filestore = 0/0 2024-03-29T02:14:19.662 INFO:teuthology.orchestra.run.smithi042.stdout: debug journal = 0/0 2024-03-29T02:14:19.662 INFO:teuthology.orchestra.run.smithi042.stdout: debug ms = 0/0 2024-03-29T02:14:19.662 INFO:teuthology.orchestra.run.smithi042.stdout: debug osd = 0/0 2024-03-29T02:14:19.663 INFO:teuthology.orchestra.run.smithi042.stdout: debug rocksdb = 10 2024-03-29T02:14:19.663 INFO:teuthology.orchestra.run.smithi042.stdout: err to stderr = True 2024-03-29T02:14:19.663 INFO:teuthology.orchestra.run.smithi042.stdout: log flush on exit = True 2024-03-29T02:14:19.663 INFO:teuthology.orchestra.run.smithi042.stdout: log to file = False 2024-03-29T02:14:19.663 INFO:teuthology.orchestra.run.smithi042.stdout: log to stderr = True 2024-03-29T02:14:19.663 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T02:14:19.663 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd full ratio = 0.9 2024-03-29T02:14:19.663 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd nearfull ratio = 0.8 2024-03-29T02:14:19.663 INFO:teuthology.orchestra.run.smithi042.stdout: osd failsafe full ratio = 0.95 2024-03-29T02:14:19.663 INFO:teuthology.orchestra.run.smithi042.stdout: osd objectstore = bluestore 2024-03-29T02:14:19.663 INFO:teuthology.orchestra.run.smithi042.stdout: osd shutdown pgref assert = True 2024-03-29T02:14:19.663 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.663 INFO:teuthology.orchestra.run.smithi042.stdout:[mgr] 2024-03-29T02:14:19.663 INFO:teuthology.orchestra.run.smithi042.stdout: debug ms = 1 2024-03-29T02:14:19.664 INFO:teuthology.orchestra.run.smithi042.stdout: debug mgr = 20 2024-03-29T02:14:19.664 INFO:teuthology.orchestra.run.smithi042.stdout: debug mon = 20 2024-03-29T02:14:19.664 INFO:teuthology.orchestra.run.smithi042.stdout: debug auth = 20 2024-03-29T02:14:19.664 INFO:teuthology.orchestra.run.smithi042.stdout: mon reweight min pgs per osd = 4 2024-03-29T02:14:19.664 INFO:teuthology.orchestra.run.smithi042.stdout: mon reweight min bytes per osd = 10 2024-03-29T02:14:19.664 INFO:teuthology.orchestra.run.smithi042.stdout: mgr/telemetry/nag = false 2024-03-29T02:14:19.664 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.664 INFO:teuthology.orchestra.run.smithi042.stdout:[mon] 2024-03-29T02:14:19.664 INFO:teuthology.orchestra.run.smithi042.stdout: debug ms = 0/0 2024-03-29T02:14:19.664 INFO:teuthology.orchestra.run.smithi042.stdout: debug mon = 0/0 2024-03-29T02:14:19.664 INFO:teuthology.orchestra.run.smithi042.stdout: debug paxos = 0/0 2024-03-29T02:14:19.664 INFO:teuthology.orchestra.run.smithi042.stdout: debug auth = 20 2024-03-29T02:14:19.664 INFO:teuthology.orchestra.run.smithi042.stdout: mon data avail warn = 5 2024-03-29T02:14:19.664 INFO:teuthology.orchestra.run.smithi042.stdout: mon mgr mkfs grace = 240 2024-03-29T02:14:19.665 INFO:teuthology.orchestra.run.smithi042.stdout: mon reweight min pgs per osd = 4 2024-03-29T02:14:19.665 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd reporter subtree level = osd 2024-03-29T02:14:19.665 INFO:teuthology.orchestra.run.smithi042.stdout: mon osd prime pg temp = true 2024-03-29T02:14:19.665 INFO:teuthology.orchestra.run.smithi042.stdout: mon reweight min bytes per osd = 10 2024-03-29T02:14:19.665 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.665 INFO:teuthology.orchestra.run.smithi042.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T02:14:19.665 INFO:teuthology.orchestra.run.smithi042.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T02:14:19.665 INFO:teuthology.orchestra.run.smithi042.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T02:14:19.665 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.665 INFO:teuthology.orchestra.run.smithi042.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T02:14:19.665 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T02:14:19.665 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T02:14:19.665 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.665 INFO:teuthology.orchestra.run.smithi042.stdout: # 1m isn't quite enough 2024-03-29T02:14:19.665 INFO:teuthology.orchestra.run.smithi042.stdout: mon_down_mkfs_grace = 2m 2024-03-29T02:14:19.666 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.666 INFO:teuthology.orchestra.run.smithi042.stdout: mon_warn_on_filestore_osds = false 2024-03-29T02:14:19.666 INFO:teuthology.orchestra.run.smithi042.stdout: osd pool default crimson = True 2024-03-29T02:14:19.666 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:19.666 INFO:teuthology.orchestra.run.smithi042.stdout:[client] 2024-03-29T02:14:19.666 INFO:teuthology.orchestra.run.smithi042.stdout: rgw cache enabled = true 2024-03-29T02:14:19.666 INFO:teuthology.orchestra.run.smithi042.stdout: rgw enable ops log = true 2024-03-29T02:14:19.666 INFO:teuthology.orchestra.run.smithi042.stdout: rgw enable usage log = true 2024-03-29T02:14:19.667 INFO:teuthology.orchestra.run.smithi042.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T02:14:19.667 INFO:teuthology.orchestra.run.smithi042.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T02:14:19.667 INFO:teuthology.orchestra.run.smithi042.stdout:[mon.a] 2024-03-29T02:14:19.667 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.0] 2024-03-29T02:14:19.667 INFO:teuthology.orchestra.run.smithi042.stdout: crimson seastar cpu cores = 0-2 2024-03-29T02:14:19.667 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.1] 2024-03-29T02:14:19.667 INFO:teuthology.orchestra.run.smithi042.stdout: crimson seastar cpu cores = 3-5 2024-03-29T02:14:19.667 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.2] 2024-03-29T02:14:19.667 INFO:teuthology.orchestra.run.smithi042.stdout: crimson seastar cpu cores = 0-2 2024-03-29T02:14:19.667 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.3] 2024-03-29T02:14:19.667 INFO:teuthology.orchestra.run.smithi042.stdout: crimson seastar cpu cores = 3-5 2024-03-29T02:14:19.668 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T02:14:19.668 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T02:14:19.732 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T02:14:19.733 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:14:19.733 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T02:14:19.747 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:14:19.747 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T02:14:19.802 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:14:19.802 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:14:19.803 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T02:14:19.803 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T02:14:19.882 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:14:19.882 DEBUG:teuthology.orchestra.run.smithi042:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T02:14:19.937 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T02:14:19.938 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T02:14:19.938 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T02:14:20.041 INFO:teuthology.orchestra.run.smithi042.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T02:14:20.044 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T02:14:20.044 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T02:14:20.044 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T02:14:20.095 INFO:teuthology.orchestra.run.smithi042.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T02:14:20.110 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T02:14:20.111 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi042.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-03-29T02:14:20.111 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T02:14:20.137 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-03-29T02:14:20.137 INFO:tasks.ceph:role: osd.0 2024-03-29T02:14:20.137 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:14:20.138 DEBUG:teuthology.orchestra.run.smithi042:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T02:14:20.204 INFO:teuthology.orchestra.run.smithi042.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T02:14:20.204 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T02:14:20.204 INFO:teuthology.orchestra.run.smithi042.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T02:14:20.204 INFO:teuthology.orchestra.run.smithi042.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T02:14:20.205 INFO:teuthology.orchestra.run.smithi042.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T02:14:20.205 INFO:teuthology.orchestra.run.smithi042.stdout: = sunit=0 swidth=0 blks 2024-03-29T02:14:20.205 INFO:teuthology.orchestra.run.smithi042.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T02:14:20.205 INFO:teuthology.orchestra.run.smithi042.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T02:14:20.205 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T02:14:20.205 INFO:teuthology.orchestra.run.smithi042.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T02:14:20.236 INFO:teuthology.orchestra.run.smithi042.stdout:Discarding blocks...Done. 2024-03-29T02:14:20.238 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi042.front.sepia.ceph.com -o noatime 2024-03-29T02:14:20.239 DEBUG:teuthology.orchestra.run.smithi042:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T02:14:20.309 DEBUG:teuthology.orchestra.run.smithi042:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T02:14:20.381 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T02:14:20.449 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-03-29T02:14:20.449 INFO:tasks.ceph:role: osd.1 2024-03-29T02:14:20.449 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:14:20.450 DEBUG:teuthology.orchestra.run.smithi042:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T02:14:20.515 INFO:teuthology.orchestra.run.smithi042.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T02:14:20.515 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T02:14:20.515 INFO:teuthology.orchestra.run.smithi042.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T02:14:20.515 INFO:teuthology.orchestra.run.smithi042.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T02:14:20.515 INFO:teuthology.orchestra.run.smithi042.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T02:14:20.516 INFO:teuthology.orchestra.run.smithi042.stdout: = sunit=0 swidth=0 blks 2024-03-29T02:14:20.516 INFO:teuthology.orchestra.run.smithi042.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T02:14:20.516 INFO:teuthology.orchestra.run.smithi042.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T02:14:20.516 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T02:14:20.516 INFO:teuthology.orchestra.run.smithi042.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T02:14:20.548 INFO:teuthology.orchestra.run.smithi042.stdout:Discarding blocks...Done. 2024-03-29T02:14:20.549 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi042.front.sepia.ceph.com -o noatime 2024-03-29T02:14:20.550 DEBUG:teuthology.orchestra.run.smithi042:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T02:14:20.620 DEBUG:teuthology.orchestra.run.smithi042:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T02:14:20.692 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T02:14:20.761 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-03-29T02:14:20.761 INFO:tasks.ceph:role: osd.2 2024-03-29T02:14:20.761 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:14:20.761 DEBUG:teuthology.orchestra.run.smithi042:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T02:14:20.828 INFO:teuthology.orchestra.run.smithi042.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T02:14:20.828 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T02:14:20.828 INFO:teuthology.orchestra.run.smithi042.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T02:14:20.828 INFO:teuthology.orchestra.run.smithi042.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T02:14:20.829 INFO:teuthology.orchestra.run.smithi042.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T02:14:20.829 INFO:teuthology.orchestra.run.smithi042.stdout: = sunit=0 swidth=0 blks 2024-03-29T02:14:20.829 INFO:teuthology.orchestra.run.smithi042.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T02:14:20.829 INFO:teuthology.orchestra.run.smithi042.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T02:14:20.829 INFO:teuthology.orchestra.run.smithi042.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T02:14:20.829 INFO:teuthology.orchestra.run.smithi042.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T02:14:20.860 INFO:teuthology.orchestra.run.smithi042.stdout:Discarding blocks...Done. 2024-03-29T02:14:20.862 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi042.front.sepia.ceph.com -o noatime 2024-03-29T02:14:20.863 DEBUG:teuthology.orchestra.run.smithi042:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T02:14:20.934 DEBUG:teuthology.orchestra.run.smithi042:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T02:14:21.004 DEBUG:teuthology.orchestra.run.smithi042:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T02:14:21.330 INFO:teuthology.orchestra.run.smithi042.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-29T02:14:21.330 INFO:teuthology.orchestra.run.smithi042.stderr:INFO 2024-03-29 02:14:21,332 seastar - Reactor backend: linux-aio 2024-03-29T02:14:21.349 INFO:teuthology.orchestra.run.smithi042.stderr:==29052==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-29T02:14:21.502 INFO:teuthology.orchestra.run.smithi042.stderr:DEBUG 2024-03-29 02:14:21,504 [shard 0:main] seastar - reactor::drain 2024-03-29T02:14:22.079 INFO:teuthology.orchestra.run.smithi042.stderr:----------------------------------------------------- 2024-03-29T02:14:22.079 INFO:teuthology.orchestra.run.smithi042.stderr:Suppressions used: 2024-03-29T02:14:22.080 INFO:teuthology.orchestra.run.smithi042.stderr: count bytes template 2024-03-29T02:14:22.080 INFO:teuthology.orchestra.run.smithi042.stderr: 1 8 MallocExtension::Register 2024-03-29T02:14:22.080 INFO:teuthology.orchestra.run.smithi042.stderr:----------------------------------------------------- 2024-03-29T02:14:22.080 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-29T02:14:22.136 INFO:teuthology.orchestra.run.smithi042.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-29T02:14:22.140 INFO:teuthology.orchestra.run.smithi042.stderr:INFO 2024-03-29 02:14:22,142 seastar - Reactor backend: linux-aio 2024-03-29T02:14:22.150 INFO:teuthology.orchestra.run.smithi042.stderr:==29048==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-29T02:14:26.831 INFO:teuthology.orchestra.run.smithi042.stderr:----------------------------------------------------- 2024-03-29T02:14:26.832 INFO:teuthology.orchestra.run.smithi042.stderr:Suppressions used: 2024-03-29T02:14:26.832 INFO:teuthology.orchestra.run.smithi042.stderr: count bytes template 2024-03-29T02:14:26.832 INFO:teuthology.orchestra.run.smithi042.stderr: 1 8 MallocExtension::Register 2024-03-29T02:14:26.832 INFO:teuthology.orchestra.run.smithi042.stderr:----------------------------------------------------- 2024-03-29T02:14:26.832 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-29T02:14:26.832 INFO:teuthology.orchestra.run.smithi042.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 4ad80ad0-2c97-4ca8-9f08-3f0610eec781 2024-03-29T02:14:26.855 DEBUG:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T02:14:26.883 DEBUG:teuthology.orchestra.run.smithi042:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T02:14:27.173 INFO:teuthology.orchestra.run.smithi042.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-29T02:14:27.174 INFO:teuthology.orchestra.run.smithi042.stderr:INFO 2024-03-29 02:14:27,175 seastar - Reactor backend: linux-aio 2024-03-29T02:14:27.179 INFO:teuthology.orchestra.run.smithi042.stderr:==29235==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-29T02:14:27.286 INFO:teuthology.orchestra.run.smithi042.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb975249 0xb847d10 0xb84928c 0xb8494a2 0xb8495f8 0xb849ac1 0x54daf 0x4de35 0x5d8bd2e 0x5d8bf56 0x5d8c16d 0x5d8c443 0x5ddac83 0x5ddae89 0x5d59970 0x5d59b4a 0x5d5a0d6 0x5b9ee91 0x5bb9f55 0x6e94504 0x6ea5cd7 0x6ea7174 0x6ea71f9 0x6ea72fb 0x6ea73f3 0x6ea7600 0x6ea77e6 0x6ea7b35 0x6ea7ed9 0x6ea8109 0x6ea8373 0x6ea8640 0x6ea885e 0x6ea8916 0x6ea89dc 0x6ea8abd 0x5a05498 0x5a055a2 0x5a0565e 0x5a0571a 0x5a057d6 0x5a05923 0x5a05aae 0x5a05e6f 0x5a228ec 0xb85a995 0xb874c2f 0xb915c2d 0xb91727b 0xb5f39e3 0xb5f435f 0x3785266 0x3785dbf 0x3625d7a 0x3feaf 0x3ff5f 0x3412444 2024-03-29T02:14:27.286 INFO:teuthology.orchestra.run.smithi042.stderr:kernel callstack: 2024-03-29T02:14:27.322 INFO:teuthology.orchestra.run.smithi042.stderr:DEBUG 2024-03-29 02:14:27,323 [shard 0:main] seastar - reactor::drain 2024-03-29T02:14:27.872 INFO:teuthology.orchestra.run.smithi042.stderr:----------------------------------------------------- 2024-03-29T02:14:27.872 INFO:teuthology.orchestra.run.smithi042.stderr:Suppressions used: 2024-03-29T02:14:27.872 INFO:teuthology.orchestra.run.smithi042.stderr: count bytes template 2024-03-29T02:14:27.872 INFO:teuthology.orchestra.run.smithi042.stderr: 1 8 MallocExtension::Register 2024-03-29T02:14:27.872 INFO:teuthology.orchestra.run.smithi042.stderr:----------------------------------------------------- 2024-03-29T02:14:27.872 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-29T02:14:27.929 INFO:teuthology.orchestra.run.smithi042.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-29T02:14:27.932 INFO:teuthology.orchestra.run.smithi042.stderr:INFO 2024-03-29 02:14:27,934 seastar - Reactor backend: linux-aio 2024-03-29T02:14:27.946 INFO:teuthology.orchestra.run.smithi042.stderr:==29233==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-29T02:14:28.011 INFO:teuthology.orchestra.run.smithi042.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb975249 0xb847d10 0xb84928c 0xb8494a2 0xb8495f8 0xb849ac1 0x54daf 0x1744d 0x174bb 0x2eab 0x4a09d 0x35b7c 0x3629e 0x5ddaceb 0x5ddae89 0x5d59970 0x5d59ea4 0x5b9ee91 0x5bb9f55 0x6e94504 0x6ea5cd7 0x6ea7174 0x6ea71f9 0x6ea72fb 0x6ea73f3 0x6ea7600 0x6ea77e6 0x6ea7b35 0x6ea7ed9 0x6ea8109 0x6ea8373 0x6ea8640 0x6ea885e 0x6ea8916 0x6ea89dc 0x6ea8abd 0x5a05498 0x5a055a2 0x5a0565e 0x5a0571a 0x5a057d6 0x5a05923 0x5a05aae 0x5a05e6f 0x5a228ec 0xb85a995 0xb874c2f 0xb915c2d 0xb91727b 0xb5f39e3 0xb5f435f 0x362658a 0x3feaf 0x3ff5f 0x3412444 2024-03-29T02:14:28.011 INFO:teuthology.orchestra.run.smithi042.stderr:kernel callstack: 2024-03-29T02:14:32.614 INFO:teuthology.orchestra.run.smithi042.stderr:----------------------------------------------------- 2024-03-29T02:14:32.614 INFO:teuthology.orchestra.run.smithi042.stderr:Suppressions used: 2024-03-29T02:14:32.614 INFO:teuthology.orchestra.run.smithi042.stderr: count bytes template 2024-03-29T02:14:32.614 INFO:teuthology.orchestra.run.smithi042.stderr: 1 8 MallocExtension::Register 2024-03-29T02:14:32.615 INFO:teuthology.orchestra.run.smithi042.stderr:----------------------------------------------------- 2024-03-29T02:14:32.615 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-29T02:14:32.615 INFO:teuthology.orchestra.run.smithi042.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 4ad80ad0-2c97-4ca8-9f08-3f0610eec781 2024-03-29T02:14:32.637 DEBUG:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T02:14:32.664 DEBUG:teuthology.orchestra.run.smithi042:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T02:14:32.956 INFO:teuthology.orchestra.run.smithi042.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-29T02:14:32.957 INFO:teuthology.orchestra.run.smithi042.stderr:INFO 2024-03-29 02:14:32,959 seastar - Reactor backend: linux-aio 2024-03-29T02:14:32.962 INFO:teuthology.orchestra.run.smithi042.stderr:==29417==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-29T02:14:33.109 INFO:teuthology.orchestra.run.smithi042.stderr:DEBUG 2024-03-29 02:14:33,111 [shard 0:main] seastar - reactor::drain 2024-03-29T02:14:33.655 INFO:teuthology.orchestra.run.smithi042.stderr:----------------------------------------------------- 2024-03-29T02:14:33.655 INFO:teuthology.orchestra.run.smithi042.stderr:Suppressions used: 2024-03-29T02:14:33.655 INFO:teuthology.orchestra.run.smithi042.stderr: count bytes template 2024-03-29T02:14:33.655 INFO:teuthology.orchestra.run.smithi042.stderr: 1 8 MallocExtension::Register 2024-03-29T02:14:33.655 INFO:teuthology.orchestra.run.smithi042.stderr:----------------------------------------------------- 2024-03-29T02:14:33.656 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-29T02:14:33.714 INFO:teuthology.orchestra.run.smithi042.stderr:WARNING: debug mode. Not for benchmarking or production 2024-03-29T02:14:33.718 INFO:teuthology.orchestra.run.smithi042.stderr:INFO 2024-03-29 02:14:33,719 seastar - Reactor backend: linux-aio 2024-03-29T02:14:33.727 INFO:teuthology.orchestra.run.smithi042.stderr:==29415==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-03-29T02:14:33.792 INFO:teuthology.orchestra.run.smithi042.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb975249 0xb847d10 0xb84928c 0xb8494a2 0xb8495f8 0xb849ac1 0x54daf 0x113ca4 0x4a0bd 0x35b7c 0x36eee 0x5d7ed9e 0x5d7ee71 0x5ddadf4 0x5d59970 0x5d59ea4 0x5b9ee91 0x5bb9f55 0x6e94504 0x6ea5cd7 0x6ea7174 0x6ea71f9 0x6ea72fb 0x6ea73f3 0x6ea7600 0x6ea77e6 0x6ea7b35 0x6ea7ed9 0x6ea8109 0x6ea8373 0x6ea8640 0x6ea885e 0x6ea8916 0x6ea89dc 0x6ea8abd 0x5a05498 0x5a055a2 0x5a0565e 0x5a0571a 0x5a057d6 0x5a05923 0x5a05aae 0x5a05e6f 0x5a228ec 0xb85a995 0xb874c2f 0xb915c2d 0xb91727b 0xb5f39e3 0xb5f435f 0x362658a 0x3feaf 0x3ff5f 0x3412444 2024-03-29T02:14:33.792 INFO:teuthology.orchestra.run.smithi042.stderr:kernel callstack: 2024-03-29T02:14:38.408 INFO:teuthology.orchestra.run.smithi042.stderr:----------------------------------------------------- 2024-03-29T02:14:38.409 INFO:teuthology.orchestra.run.smithi042.stderr:Suppressions used: 2024-03-29T02:14:38.409 INFO:teuthology.orchestra.run.smithi042.stderr: count bytes template 2024-03-29T02:14:38.409 INFO:teuthology.orchestra.run.smithi042.stderr: 1 8 MallocExtension::Register 2024-03-29T02:14:38.409 INFO:teuthology.orchestra.run.smithi042.stderr:----------------------------------------------------- 2024-03-29T02:14:38.409 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-29T02:14:38.409 INFO:teuthology.orchestra.run.smithi042.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 4ad80ad0-2c97-4ca8-9f08-3f0610eec781 2024-03-29T02:14:38.433 DEBUG:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T02:14:38.462 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T02:14:38.462 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:14:38.463 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T02:14:38.530 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:14:38.530 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T02:14:38.600 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:14:38.600 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T02:14:38.668 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:14:38.669 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T02:14:38.736 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:14:38.736 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T02:14:38.791 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T02:14:38.791 DEBUG:teuthology.orchestra.run.smithi042:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T02:14:38.856 INFO:teuthology.orchestra.run.smithi042.stdout:[mgr.x] 2024-03-29T02:14:38.856 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQD8IwZm3siPAhAAXW1A7QighTqv8ThTMfGWNQ== 2024-03-29T02:14:38.856 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.0] 2024-03-29T02:14:38.856 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQD+IwZmdCZ1EhAAUMvpofa6b35/IYhsFE81AQ== 2024-03-29T02:14:38.856 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.1] 2024-03-29T02:14:38.856 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAEJAZmTAZuBhAAGhQ1MlxJq4r+aW4YmHL/bQ== 2024-03-29T02:14:38.856 INFO:teuthology.orchestra.run.smithi042.stdout:[osd.2] 2024-03-29T02:14:38.856 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQAJJAZmFNPcNBAAbbMOwakFr/1X6Qd+LIX0PQ== 2024-03-29T02:14:38.856 INFO:teuthology.orchestra.run.smithi042.stdout:[client.0] 2024-03-29T02:14:38.856 INFO:teuthology.orchestra.run.smithi042.stdout: key = AQD8IwZmuBzIBRAA2LYYmQfdJNsNX24bohxh/Q== 2024-03-29T02:14:38.857 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T02:14:38.959 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T02:14:39.020 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T02:14:39.081 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T02:14:39.142 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T02:14:39.206 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T02:14:39.206 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T02:14:39.234 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T02:14:39.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:39.345+0000 7f37be491a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:39.345 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:39.346+0000 7f37be491a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:39.358 DEBUG:teuthology.orchestra.run.smithi042:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T02:14:39.385 DEBUG:teuthology.orchestra.run.smithi042:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T02:14:39.439 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T02:14:39.440 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T02:14:39.441 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T02:14:39.482 INFO:tasks.ceph.mon.a:Started 2024-03-29T02:14:39.483 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T02:14:39.483 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T02:14:39.483 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T02:14:39.485 INFO:tasks.ceph.mgr.x:Started 2024-03-29T02:14:39.485 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T02:14:39.485 DEBUG:teuthology.orchestra.run.smithi042:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T02:14:39.568 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-03-29T02:14:39.569+0000 7f311d46aa80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:39.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-03-29T02:14:39.569+0000 7f311d46aa80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:39.569 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-03-29T02:14:39.569+0000 7f311d46aa80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:39.570 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:39.571+0000 7fde81c8e000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:39.599 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:39.595+0000 7fb778024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:39.601 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:39.602+0000 7fb778024640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:39.672 INFO:teuthology.orchestra.run.smithi042.stderr:adjusted tunables profile to default 2024-03-29T02:14:39.683 INFO:tasks.ceph:check_enable_crimson: True 2024-03-29T02:14:39.683 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-03-29T02:14:39.683 DEBUG:teuthology.orchestra.run.smithi042:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-03-29T02:14:39.773 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:39.774+0000 7fde81c8e000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:39.773 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:39.774+0000 7fde81c8e000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:39.799 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:39.795+0000 7ff43bb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:39.801 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:39.802+0000 7ff43bb9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:39.921 INFO:tasks.ceph.mon.a.smithi042.stderr:2024-03-29T02:14:39.922+0000 7f31141f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-03-29T02:14:39.921 INFO:tasks.ceph.mon.a.smithi042.stderr:Please be aware that this feature is experimental, untested, 2024-03-29T02:14:39.921 INFO:tasks.ceph.mon.a.smithi042.stderr:unsupported, and may result in data corruption, data loss, 2024-03-29T02:14:39.921 INFO:tasks.ceph.mon.a.smithi042.stderr:and/or irreparable damage to your cluster. Do not use 2024-03-29T02:14:39.921 INFO:tasks.ceph.mon.a.smithi042.stderr:feature with important data. 2024-03-29T02:14:39.921 INFO:tasks.ceph.mon.a.smithi042.stderr: 2024-03-29T02:14:40.303 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:40.304+0000 7fde81c8e000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T02:14:40.359 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:40.360+0000 7fde81c8e000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T02:14:40.685 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T02:14:40.685 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:14:40.685 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T02:14:40.712 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:14:40.712 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T02:14:40.779 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:14:40.779 DEBUG:teuthology.orchestra.run.smithi042:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T02:14:40.846 DEBUG:teuthology.orchestra.run.smithi042:> sudo ceph --cluster ceph osd new 7b4b3877-6714-430b-8f55-9f260ee9795a 0 2024-03-29T02:14:40.908 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:40.909+0000 7fde81c8e000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T02:14:40.968 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:40.969+0000 7fde81c8e000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T02:14:40.998 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:40.993+0000 7ff15d3e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:41.000 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:41.001+0000 7ff15d3e0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:41.015 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:41.016+0000 7fde81c8e000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T02:14:41.071 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:41.073+0000 7fde81c8e000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T02:14:41.079 INFO:teuthology.orchestra.run.smithi042.stdout:0 2024-03-29T02:14:41.091 DEBUG:teuthology.orchestra.run.smithi042:> sudo ceph --cluster ceph osd new 60324998-2a1d-46a3-af70-576cc63a376e 1 2024-03-29T02:14:41.117 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:41.118+0000 7fde81c8e000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T02:14:41.228 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:41.224+0000 7f2f78aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:41.230 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:41.231+0000 7f2f78aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:41.243 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:41.244+0000 7fde81c8e000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T02:14:41.309 INFO:teuthology.orchestra.run.smithi042.stdout:1 2024-03-29T02:14:41.317 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:41.318+0000 7fde81c8e000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T02:14:41.320 DEBUG:teuthology.orchestra.run.smithi042:> sudo ceph --cluster ceph osd new d8902125-a76f-4ece-bfff-2ae8f81a3aec 2 2024-03-29T02:14:41.367 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:41.367+0000 7fde81c8e000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T02:14:41.424 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:41.425+0000 7fde81c8e000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T02:14:41.431 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:41.427+0000 7f08f41f1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:41.434 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:41.435+0000 7f08f41f1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:41.474 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:41.475+0000 7fde81c8e000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T02:14:41.512 INFO:teuthology.orchestra.run.smithi042.stdout:2 2024-03-29T02:14:41.523 DEBUG:teuthology.orchestra.run.smithi042:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-03-29T02:14:41.554 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T02:14:41.555 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-03-29T02:14:41.597 INFO:tasks.ceph.osd.0:Started 2024-03-29T02:14:41.597 DEBUG:teuthology.orchestra.run.smithi042:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-03-29T02:14:41.629 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T02:14:41.629 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-03-29T02:14:41.671 INFO:tasks.ceph.osd.1:Started 2024-03-29T02:14:41.671 DEBUG:teuthology.orchestra.run.smithi042:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-03-29T02:14:41.704 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T02:14:41.705 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-03-29T02:14:41.748 INFO:tasks.ceph.osd.2:Started 2024-03-29T02:14:41.749 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T02:14:41.875 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:41.872+0000 7f02b4cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:41.878 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:41.879+0000 7f02b4cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:41.910 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:41.911+0000 7fde81c8e000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T02:14:41.948 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:41.948 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":6,"fsid":"4ad80ad0-2c97-4ca8-9f08-3f0610eec781","created":"2024-03-29T02:14:39.587895+0000","modified":"2024-03-29T02:14:41.513575+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T02:14:41.513575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"7b4b3877-6714-430b-8f55-9f260ee9795a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"60324998-2a1d-46a3-af70-576cc63a376e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8902125-a76f-4ece-bfff-2ae8f81a3aec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T02:14:41.961 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T02:14:41.961 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T02:14:42.099 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:42.100+0000 7fde81c8e000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T02:14:42.226 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:42.227+0000 7fde81c8e000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T02:14:42.264 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T02:14:42.276 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:42.277+0000 7fde81c8e000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T02:14:42.368 INFO:teuthology.misc.health.smithi042.stderr:2024-03-29T02:14:42.365+0000 7f47bdcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:42.370 INFO:teuthology.misc.health.smithi042.stderr:2024-03-29T02:14:42.371+0000 7f47bdcad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:42.402 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:42.403+0000 7fde81c8e000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T02:14:42.440 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-29T02:14:42.440 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":6,"fsid":"4ad80ad0-2c97-4ca8-9f08-3f0610eec781","created":"2024-03-29T02:14:39.587895+0000","modified":"2024-03-29T02:14:41.513575+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T02:14:41.513575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"7b4b3877-6714-430b-8f55-9f260ee9795a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"60324998-2a1d-46a3-af70-576cc63a376e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d8902125-a76f-4ece-bfff-2ae8f81a3aec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T02:14:42.451 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-03-29T02:14:42.510 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:42.510+0000 7fde81c8e000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T02:14:42.684 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:42.685+0000 7fde81c8e000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T02:14:42.732 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:42.733+0000 7fde81c8e000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T02:14:42.904 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:42.905+0000 7fde81c8e000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T02:14:43.776 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:43.777+0000 7fde81c8e000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T02:14:44.082 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:44.083+0000 7fde81c8e000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T02:14:44.143 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:44.144+0000 7fde81c8e000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T02:14:44.280 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:44.280+0000 7fde81c8e000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T02:14:45.289 INFO:tasks.ceph.mgr.x.smithi042.stderr:2024-03-29T02:14:45.290+0000 7fde212dc640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-29T02:14:48.754 DEBUG:teuthology.orchestra.run.smithi042:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T02:14:48.871 INFO:teuthology.misc.health.smithi042.stderr:2024-03-29T02:14:48.869+0000 7f3a9806d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:48.874 INFO:teuthology.misc.health.smithi042.stderr:2024-03-29T02:14:48.875+0000 7f3a9806d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:49.072 INFO:teuthology.misc.health.smithi042.stdout: 2024-03-29T02:14:49.073 INFO:teuthology.misc.health.smithi042.stdout:{"epoch":9,"fsid":"4ad80ad0-2c97-4ca8-9f08-3f0610eec781","created":"2024-03-29T02:14:39.587895+0000","modified":"2024-03-29T02:14:48.297277+0000","last_up_change":"2024-03-29T02:14:48.297277+0000","last_in_change":"2024-03-29T02:14:41.513575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"7b4b3877-6714-430b-8f55-9f260ee9795a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6809","nonce":2079228219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":2079228219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":2079228219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6811","nonce":2079228219}]},"public_addr":"172.21.15.42:6809/2079228219","cluster_addr":"172.21.15.42:6810/2079228219","heartbeat_back_addr":"172.21.15.42:6812/2079228219","heartbeat_front_addr":"172.21.15.42:6811/2079228219","state":["exists","up"]},{"osd":1,"uuid":"60324998-2a1d-46a3-af70-576cc63a376e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6805","nonce":607262436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":607262436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6807","nonce":607262436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":607262436}]},"public_addr":"172.21.15.42:6805/607262436","cluster_addr":"172.21.15.42:6806/607262436","heartbeat_back_addr":"172.21.15.42:6807/607262436","heartbeat_front_addr":"172.21.15.42:6808/607262436","state":["exists","up"]},{"osd":2,"uuid":"d8902125-a76f-4ece-bfff-2ae8f81a3aec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6803","nonce":1893685367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":1893685367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6813","nonce":1893685367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":1893685367}]},"public_addr":"172.21.15.42:6803/1893685367","cluster_addr":"172.21.15.42:6802/1893685367","heartbeat_back_addr":"172.21.15.42:6813/1893685367","heartbeat_front_addr":"172.21.15.42:6804/1893685367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T02:14:49.084 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-03-29T02:14:49.084 INFO:tasks.ceph:Creating RBD pool 2024-03-29T02:14:49.084 DEBUG:teuthology.orchestra.run.smithi042:> sudo ceph --cluster ceph osd pool create rbd 8 2024-03-29T02:14:49.198 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:49.192+0000 7f8021b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:49.200 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:49.201+0000 7f8021b9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:50.303 INFO:teuthology.orchestra.run.smithi042.stderr:pool 'rbd' created 2024-03-29T02:14:50.331 DEBUG:teuthology.orchestra.run.smithi042:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-03-29T02:14:50.471 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:50.462+0000 7fe98b060640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:50.473 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:50.474+0000 7fe98b060640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:51.306 INFO:teuthology.orchestra.run.smithi042.stderr:enabled application 'rbd' on pool 'rbd' 2024-03-29T02:14:51.323 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-29T02:14:51.324 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-29T02:14:51.324 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-29T02:14:51.324 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-29T02:14:51.324 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-29T02:14:51.484 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:51.481+0000 7f193719d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:51.485 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:51.487+0000 7f193719d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:51.730 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:51.740 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":4,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6800","nonce":3425084693},{"type":"v1","addr":"172.21.15.42:6801","nonce":3425084693}]},"active_addr":"172.21.15.42:6801/3425084693","active_change":"2024-03-29T02:14:44.283420+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.42:0","nonce":428829588}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.42:0","nonce":3161736641}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.42:0","nonce":3522285391}]}]} 2024-03-29T02:14:51.745 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-29T02:14:51.745 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-29T02:14:51.745 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T02:14:51.859 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:51.857+0000 7f6b63fff640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:51.861 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:51.862+0000 7f6b62d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:52.055 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:52.055 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":12,"fsid":"4ad80ad0-2c97-4ca8-9f08-3f0610eec781","created":"2024-03-29T02:14:39.587895+0000","modified":"2024-03-29T02:14:51.307656+0000","last_up_change":"2024-03-29T02:14:48.297277+0000","last_in_change":"2024-03-29T02:14:41.513575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T02:14:48.305426+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-29T02:14:49.408690+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4b3877-6714-430b-8f55-9f260ee9795a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6809","nonce":2079228219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":2079228219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":2079228219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6811","nonce":2079228219}]},"public_addr":"172.21.15.42:6809/2079228219","cluster_addr":"172.21.15.42:6810/2079228219","heartbeat_back_addr":"172.21.15.42:6812/2079228219","heartbeat_front_addr":"172.21.15.42:6811/2079228219","state":["exists","up"]},{"osd":1,"uuid":"60324998-2a1d-46a3-af70-576cc63a376e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6805","nonce":607262436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":607262436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6807","nonce":607262436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":607262436}]},"public_addr":"172.21.15.42:6805/607262436","cluster_addr":"172.21.15.42:6806/607262436","heartbeat_back_addr":"172.21.15.42:6807/607262436","heartbeat_front_addr":"172.21.15.42:6808/607262436","state":["exists","up"]},{"osd":2,"uuid":"d8902125-a76f-4ece-bfff-2ae8f81a3aec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6803","nonce":1893685367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":1893685367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6813","nonce":1893685367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":1893685367}]},"public_addr":"172.21.15.42:6803/1893685367","cluster_addr":"172.21.15.42:6802/1893685367","heartbeat_back_addr":"172.21.15.42:6813/1893685367","heartbeat_front_addr":"172.21.15.42:6804/1893685367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T02:14:52.067 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-29T02:14:52.067 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T02:14:52.182 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:52.180+0000 7f172c26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:52.184 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:52.185+0000 7f172c26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:52.395 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:52.395 INFO:teuthology.orchestra.run.smithi042.stdout:{"epoch":13,"fsid":"4ad80ad0-2c97-4ca8-9f08-3f0610eec781","created":"2024-03-29T02:14:39.587895+0000","modified":"2024-03-29T02:14:52.311913+0000","last_up_change":"2024-03-29T02:14:48.297277+0000","last_in_change":"2024-03-29T02:14:41.513575+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-29T02:14:48.305426+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-03-29T02:14:49.408690+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b4b3877-6714-430b-8f55-9f260ee9795a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6809","nonce":2079228219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6810","nonce":2079228219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6812","nonce":2079228219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6811","nonce":2079228219}]},"public_addr":"172.21.15.42:6809/2079228219","cluster_addr":"172.21.15.42:6810/2079228219","heartbeat_back_addr":"172.21.15.42:6812/2079228219","heartbeat_front_addr":"172.21.15.42:6811/2079228219","state":["exists","up"]},{"osd":1,"uuid":"60324998-2a1d-46a3-af70-576cc63a376e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6805","nonce":607262436}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6806","nonce":607262436}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6807","nonce":607262436}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6808","nonce":607262436}]},"public_addr":"172.21.15.42:6805/607262436","cluster_addr":"172.21.15.42:6806/607262436","heartbeat_back_addr":"172.21.15.42:6807/607262436","heartbeat_front_addr":"172.21.15.42:6808/607262436","state":["exists","up"]},{"osd":2,"uuid":"d8902125-a76f-4ece-bfff-2ae8f81a3aec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6803","nonce":1893685367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6802","nonce":1893685367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6813","nonce":1893685367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.42:6804","nonce":1893685367}]},"public_addr":"172.21.15.42:6803/1893685367","cluster_addr":"172.21.15.42:6802/1893685367","heartbeat_back_addr":"172.21.15.42:6813/1893685367","heartbeat_front_addr":"172.21.15.42:6804/1893685367","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T02:14:52.408 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-29T02:14:52.409 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-29T02:14:52.409 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-29T02:14:52.528 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:52.525+0000 7fa0cff5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:52.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:52.525+0000 7f304486d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:52.529 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:52.525+0000 7fe28e19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:52.534 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:52.532+0000 7fe28e19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:52.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:52.532+0000 7fa0cff5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:52.535 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:52.532+0000 7f304486d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:52.559 INFO:teuthology.orchestra.run.smithi042.stdout:34359738373 2024-03-29T02:14:52.559 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-29T02:14:52.562 INFO:teuthology.orchestra.run.smithi042.stdout:34359738373 2024-03-29T02:14:52.562 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-29T02:14:52.565 INFO:teuthology.orchestra.run.smithi042.stdout:38654705668 2024-03-29T02:14:52.566 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-29T02:14:52.675 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:52.671+0000 7fb502f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:52.677 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:52.678+0000 7fb502f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:52.725 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:52.721+0000 7fd72a19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:52.727 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:52.728+0000 7fd72a19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:52.743 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:52.739+0000 7f4d8e6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:52.745 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:52.746+0000 7f4d8e6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:52.900 INFO:teuthology.orchestra.run.smithi042.stdout:0 2024-03-29T02:14:52.913 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.1 2024-03-29T02:14:52.929 INFO:teuthology.orchestra.run.smithi042.stdout:0 2024-03-29T02:14:52.942 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.2 2024-03-29T02:14:52.947 INFO:teuthology.orchestra.run.smithi042.stdout:0 2024-03-29T02:14:52.959 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.0 2024-03-29T02:14:53.914 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-29T02:14:53.943 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-29T02:14:53.962 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-29T02:14:54.047 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:54.041+0000 7fd175c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:54.050 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:54.051+0000 7fd175c6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:54.087 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:54.084+0000 7fc74bc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:54.088 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:54.084+0000 7fea4e6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:54.090 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:54.091+0000 7fc74bc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:54.091 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:54.091+0000 7fea4e6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:54.258 INFO:teuthology.orchestra.run.smithi042.stdout:0 2024-03-29T02:14:54.271 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.1 2024-03-29T02:14:54.296 INFO:teuthology.orchestra.run.smithi042.stdout:0 2024-03-29T02:14:54.302 INFO:teuthology.orchestra.run.smithi042.stdout:0 2024-03-29T02:14:54.311 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.0 2024-03-29T02:14:54.316 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.2 2024-03-29T02:14:55.272 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-29T02:14:55.312 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-29T02:14:55.318 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-29T02:14:55.408 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:55.402+0000 7f2c5dead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:55.410 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:55.411+0000 7f2c5dead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:55.442 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:55.439+0000 7f22ed06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:55.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:55.440+0000 7fd34b617640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:55.444 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:55.446+0000 7f22ed06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:55.446 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:55.447+0000 7fd34b617640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:55.610 INFO:teuthology.orchestra.run.smithi042.stdout:0 2024-03-29T02:14:55.623 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.1 2024-03-29T02:14:55.644 INFO:teuthology.orchestra.run.smithi042.stdout:0 2024-03-29T02:14:55.651 INFO:teuthology.orchestra.run.smithi042.stdout:0 2024-03-29T02:14:55.657 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.0 2024-03-29T02:14:55.664 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.2 2024-03-29T02:14:56.624 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-29T02:14:56.658 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-29T02:14:56.664 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-29T02:14:56.761 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:56.755+0000 7fbdf986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:56.763 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:56.764+0000 7fbdf986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:56.796 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:56.792+0000 7f225719d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:56.798 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:56.799+0000 7f225719d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:56.803 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:56.798+0000 7fdeac217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:56.807 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:56.808+0000 7fdeac217640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:56.971 INFO:teuthology.orchestra.run.smithi042.stdout:0 2024-03-29T02:14:56.989 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.1 2024-03-29T02:14:57.002 INFO:teuthology.orchestra.run.smithi042.stdout:0 2024-03-29T02:14:57.003 INFO:teuthology.orchestra.run.smithi042.stdout:0 2024-03-29T02:14:57.015 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.0 2024-03-29T02:14:57.016 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.2 2024-03-29T02:14:57.990 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-29T02:14:58.016 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-29T02:14:58.017 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-29T02:14:58.125 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:58.119+0000 7f4cb0e60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:58.127 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:58.128+0000 7f4cb0e60640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:58.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:58.156+0000 7fda6de6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:58.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:58.157+0000 7ff172114640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:58.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:58.163+0000 7fda6de6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:58.162 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:58.163+0000 7ff172114640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:58.339 INFO:teuthology.orchestra.run.smithi042.stdout:34359738378 2024-03-29T02:14:58.356 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738378 for osd.1 2024-03-29T02:14:58.356 DEBUG:teuthology.parallel:result is None 2024-03-29T02:14:58.372 INFO:teuthology.orchestra.run.smithi042.stdout:38654705673 2024-03-29T02:14:58.383 INFO:teuthology.orchestra.run.smithi042.stdout:34359738377 2024-03-29T02:14:58.386 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705673 for osd.2 2024-03-29T02:14:58.386 DEBUG:teuthology.parallel:result is None 2024-03-29T02:14:58.395 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738377 for osd.0 2024-03-29T02:14:58.395 DEBUG:teuthology.parallel:result is None 2024-03-29T02:14:58.396 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-29T02:14:58.396 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T02:14:58.548 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:58.546+0000 7f86b38ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:58.550 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:58.551+0000 7f86b38ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:58.742 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:58.742 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-03-29T02:14:58.754 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-03-29T02:14:58.326094+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80932,"kb_used_data":436,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034588,"statfs":{"total":289910292480,"available":289827418112,"internally_reserved":0,"allocated":446464,"data_stored":1257430,"data_compressed":59800,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-29T02:14:52.341748+0000","last_change":"2024-03-29T02:14:51.439127+0000","last_active":"2024-03-29T02:14:52.341748+0000","last_peered":"2024-03-29T02:14:52.341748+0000","last_clean":"2024-03-29T02:14:52.341748+0000","last_became_active":"2024-03-29T02:14:51.437007+0000","last_became_peered":"2024-03-29T02:14:51.437007+0000","last_unstale":"2024-03-29T02:14:52.341748+0000","last_undegraded":"2024-03-29T02:14:52.341748+0000","last_fullsized":"2024-03-29T02:14:52.341748+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_clean_scrub_stamp":"2024-03-29T02:14:50.303520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-29T02:14:52.342348+0000","last_change":"2024-03-29T02:14:51.411461+0000","last_active":"2024-03-29T02:14:52.342348+0000","last_peered":"2024-03-29T02:14:52.342348+0000","last_clean":"2024-03-29T02:14:52.342348+0000","last_became_active":"2024-03-29T02:14:51.409328+0000","last_became_peered":"2024-03-29T02:14:51.409328+0000","last_unstale":"2024-03-29T02:14:52.342348+0000","last_undegraded":"2024-03-29T02:14:52.342348+0000","last_fullsized":"2024-03-29T02:14:52.342348+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_clean_scrub_stamp":"2024-03-29T02:14:50.303520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-29T02:14:52.339869+0000","last_change":"2024-03-29T02:14:51.400100+0000","last_active":"2024-03-29T02:14:52.339869+0000","last_peered":"2024-03-29T02:14:52.339869+0000","last_clean":"2024-03-29T02:14:52.339869+0000","last_became_active":"2024-03-29T02:14:51.394724+0000","last_became_peered":"2024-03-29T02:14:51.394724+0000","last_unstale":"2024-03-29T02:14:52.339869+0000","last_undegraded":"2024-03-29T02:14:52.339869+0000","last_fullsized":"2024-03-29T02:14:52.339869+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_clean_scrub_stamp":"2024-03-29T02:14:50.303520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-29T02:14:52.342736+0000","last_change":"2024-03-29T02:14:51.429077+0000","last_active":"2024-03-29T02:14:52.342736+0000","last_peered":"2024-03-29T02:14:52.342736+0000","last_clean":"2024-03-29T02:14:52.342736+0000","last_became_active":"2024-03-29T02:14:51.427123+0000","last_became_peered":"2024-03-29T02:14:51.427123+0000","last_unstale":"2024-03-29T02:14:52.342736+0000","last_undegraded":"2024-03-29T02:14:52.342736+0000","last_fullsized":"2024-03-29T02:14:52.342736+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_clean_scrub_stamp":"2024-03-29T02:14:50.303520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-29T02:14:51.392198+0000","last_change":"2024-03-29T02:14:51.392479+0000","last_active":"2024-03-29T02:14:51.392198+0000","last_peered":"2024-03-29T02:14:51.392198+0000","last_clean":"2024-03-29T02:14:50.303520+0000","last_became_active":"2024-03-29T02:14:51.390400+0000","last_became_peered":"2024-03-29T02:14:51.390400+0000","last_unstale":"2024-03-29T02:14:51.392198+0000","last_undegraded":"2024-03-29T02:14:51.392198+0000","last_fullsized":"2024-03-29T02:14:51.392198+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_clean_scrub_stamp":"2024-03-29T02:14:50.303520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-29T02:14:53.341088+0000","last_change":"2024-03-29T02:14:51.397874+0000","last_active":"2024-03-29T02:14:53.341088+0000","last_peered":"2024-03-29T02:14:53.341088+0000","last_clean":"2024-03-29T02:14:53.341088+0000","last_became_active":"2024-03-29T02:14:51.395828+0000","last_became_peered":"2024-03-29T02:14:51.395828+0000","last_unstale":"2024-03-29T02:14:53.341088+0000","last_undegraded":"2024-03-29T02:14:53.341088+0000","last_fullsized":"2024-03-29T02:14:53.341088+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_clean_scrub_stamp":"2024-03-29T02:14:50.303520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-29T02:14:53.339352+0000","last_change":"2024-03-29T02:14:51.373623+0000","last_active":"2024-03-29T02:14:53.339352+0000","last_peered":"2024-03-29T02:14:53.339352+0000","last_clean":"2024-03-29T02:14:53.339352+0000","last_became_active":"2024-03-29T02:14:51.370737+0000","last_became_peered":"2024-03-29T02:14:51.370737+0000","last_unstale":"2024-03-29T02:14:53.339352+0000","last_undegraded":"2024-03-29T02:14:53.339352+0000","last_fullsized":"2024-03-29T02:14:53.339352+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_clean_scrub_stamp":"2024-03-29T02:14:50.303520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-29T02:14:52.342539+0000","last_change":"2024-03-29T02:14:51.369062+0000","last_active":"2024-03-29T02:14:52.342539+0000","last_peered":"2024-03-29T02:14:52.342539+0000","last_clean":"2024-03-29T02:14:52.342539+0000","last_became_active":"2024-03-29T02:14:51.366639+0000","last_became_peered":"2024-03-29T02:14:51.366639+0000","last_unstale":"2024-03-29T02:14:52.342539+0000","last_undegraded":"2024-03-29T02:14:52.342539+0000","last_fullsized":"2024-03-29T02:14:52.342539+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_clean_scrub_stamp":"2024-03-29T02:14:50.303520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-29T02:14:52.338790+0000","last_change":"2024-03-29T02:14:50.372132+0000","last_active":"2024-03-29T02:14:52.338790+0000","last_peered":"2024-03-29T02:14:52.338790+0000","last_clean":"2024-03-29T02:14:52.338790+0000","last_became_active":"2024-03-29T02:14:50.366175+0000","last_became_peered":"2024-03-29T02:14:50.366175+0000","last_unstale":"2024-03-29T02:14:52.338790+0000","last_undegraded":"2024-03-29T02:14:52.338790+0000","last_fullsized":"2024-03-29T02:14:52.338790+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:14:49.300701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:14:49.300701+0000","last_clean_scrub_stamp":"2024-03-29T02:14:49.300701+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":26529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":616897,"data_compressed":29900,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":614004,"data_compressed":29900,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-29T02:14:58.755 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-29T02:14:58.867 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:58.865+0000 7f6a1b11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:58.869 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:58.870+0000 7f6a1b11d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:59.059 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:59.059 INFO:teuthology.orchestra.run.smithi042.stderr:dumped all 2024-03-29T02:14:59.071 INFO:teuthology.orchestra.run.smithi042.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-03-29T02:14:58.326094+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80932,"kb_used_data":436,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034588,"statfs":{"total":289910292480,"available":289827418112,"internally_reserved":0,"allocated":446464,"data_stored":1257430,"data_compressed":59800,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-29T02:14:52.341748+0000","last_change":"2024-03-29T02:14:51.439127+0000","last_active":"2024-03-29T02:14:52.341748+0000","last_peered":"2024-03-29T02:14:52.341748+0000","last_clean":"2024-03-29T02:14:52.341748+0000","last_became_active":"2024-03-29T02:14:51.437007+0000","last_became_peered":"2024-03-29T02:14:51.437007+0000","last_unstale":"2024-03-29T02:14:52.341748+0000","last_undegraded":"2024-03-29T02:14:52.341748+0000","last_fullsized":"2024-03-29T02:14:52.341748+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_clean_scrub_stamp":"2024-03-29T02:14:50.303520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-29T02:14:52.342348+0000","last_change":"2024-03-29T02:14:51.411461+0000","last_active":"2024-03-29T02:14:52.342348+0000","last_peered":"2024-03-29T02:14:52.342348+0000","last_clean":"2024-03-29T02:14:52.342348+0000","last_became_active":"2024-03-29T02:14:51.409328+0000","last_became_peered":"2024-03-29T02:14:51.409328+0000","last_unstale":"2024-03-29T02:14:52.342348+0000","last_undegraded":"2024-03-29T02:14:52.342348+0000","last_fullsized":"2024-03-29T02:14:52.342348+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_clean_scrub_stamp":"2024-03-29T02:14:50.303520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-29T02:14:52.339869+0000","last_change":"2024-03-29T02:14:51.400100+0000","last_active":"2024-03-29T02:14:52.339869+0000","last_peered":"2024-03-29T02:14:52.339869+0000","last_clean":"2024-03-29T02:14:52.339869+0000","last_became_active":"2024-03-29T02:14:51.394724+0000","last_became_peered":"2024-03-29T02:14:51.394724+0000","last_unstale":"2024-03-29T02:14:52.339869+0000","last_undegraded":"2024-03-29T02:14:52.339869+0000","last_fullsized":"2024-03-29T02:14:52.339869+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_clean_scrub_stamp":"2024-03-29T02:14:50.303520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-29T02:14:52.342736+0000","last_change":"2024-03-29T02:14:51.429077+0000","last_active":"2024-03-29T02:14:52.342736+0000","last_peered":"2024-03-29T02:14:52.342736+0000","last_clean":"2024-03-29T02:14:52.342736+0000","last_became_active":"2024-03-29T02:14:51.427123+0000","last_became_peered":"2024-03-29T02:14:51.427123+0000","last_unstale":"2024-03-29T02:14:52.342736+0000","last_undegraded":"2024-03-29T02:14:52.342736+0000","last_fullsized":"2024-03-29T02:14:52.342736+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_clean_scrub_stamp":"2024-03-29T02:14:50.303520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-03-29T02:14:51.392198+0000","last_change":"2024-03-29T02:14:51.392479+0000","last_active":"2024-03-29T02:14:51.392198+0000","last_peered":"2024-03-29T02:14:51.392198+0000","last_clean":"2024-03-29T02:14:50.303520+0000","last_became_active":"2024-03-29T02:14:51.390400+0000","last_became_peered":"2024-03-29T02:14:51.390400+0000","last_unstale":"2024-03-29T02:14:51.392198+0000","last_undegraded":"2024-03-29T02:14:51.392198+0000","last_fullsized":"2024-03-29T02:14:51.392198+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_clean_scrub_stamp":"2024-03-29T02:14:50.303520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-29T02:14:53.341088+0000","last_change":"2024-03-29T02:14:51.397874+0000","last_active":"2024-03-29T02:14:53.341088+0000","last_peered":"2024-03-29T02:14:53.341088+0000","last_clean":"2024-03-29T02:14:53.341088+0000","last_became_active":"2024-03-29T02:14:51.395828+0000","last_became_peered":"2024-03-29T02:14:51.395828+0000","last_unstale":"2024-03-29T02:14:53.341088+0000","last_undegraded":"2024-03-29T02:14:53.341088+0000","last_fullsized":"2024-03-29T02:14:53.341088+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_clean_scrub_stamp":"2024-03-29T02:14:50.303520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-29T02:14:53.339352+0000","last_change":"2024-03-29T02:14:51.373623+0000","last_active":"2024-03-29T02:14:53.339352+0000","last_peered":"2024-03-29T02:14:53.339352+0000","last_clean":"2024-03-29T02:14:53.339352+0000","last_became_active":"2024-03-29T02:14:51.370737+0000","last_became_peered":"2024-03-29T02:14:51.370737+0000","last_unstale":"2024-03-29T02:14:53.339352+0000","last_undegraded":"2024-03-29T02:14:53.339352+0000","last_fullsized":"2024-03-29T02:14:53.339352+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_clean_scrub_stamp":"2024-03-29T02:14:50.303520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-29T02:14:52.342539+0000","last_change":"2024-03-29T02:14:51.369062+0000","last_active":"2024-03-29T02:14:52.342539+0000","last_peered":"2024-03-29T02:14:52.342539+0000","last_clean":"2024-03-29T02:14:52.342539+0000","last_became_active":"2024-03-29T02:14:51.366639+0000","last_became_peered":"2024-03-29T02:14:51.366639+0000","last_unstale":"2024-03-29T02:14:52.342539+0000","last_undegraded":"2024-03-29T02:14:52.342539+0000","last_fullsized":"2024-03-29T02:14:52.342539+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:14:50.303520+0000","last_clean_scrub_stamp":"2024-03-29T02:14:50.303520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-03-29T02:14:52.338790+0000","last_change":"2024-03-29T02:14:50.372132+0000","last_active":"2024-03-29T02:14:52.338790+0000","last_peered":"2024-03-29T02:14:52.338790+0000","last_clean":"2024-03-29T02:14:52.338790+0000","last_became_active":"2024-03-29T02:14:50.366175+0000","last_became_peered":"2024-03-29T02:14:50.366175+0000","last_unstale":"2024-03-29T02:14:52.338790+0000","last_undegraded":"2024-03-29T02:14:52.338790+0000","last_fullsized":"2024-03-29T02:14:52.338790+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-29T02:14:49.300701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-29T02:14:49.300701+0000","last_clean_scrub_stamp":"2024-03-29T02:14:49.300701+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705673,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26932,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344908,"statfs":{"total":96636764160,"available":96609185792,"internally_reserved":0,"allocated":110592,"data_stored":26529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":616897,"data_compressed":29900,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":614004,"data_compressed":29900,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-03-29T02:14:59.072 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-29T02:14:59.072 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-29T02:14:59.072 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-29T02:14:59.073 DEBUG:teuthology.orchestra.run.smithi042:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-29T02:14:59.185 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:59.184+0000 7fc125b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:59.186 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:14:59.188+0000 7fc125b5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:14:59.426 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:14:59.427 INFO:teuthology.orchestra.run.smithi042.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-29T02:14:59.441 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-29T02:14:59.442 INFO:teuthology.run_tasks:Running task cbt... 2024-03-29T02:14:59.456 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi042.front.sepia.ceph.com', 'osds': ['ubuntu@smithi042.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi042.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi042.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi042.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-03-29T02:14:59.456 DEBUG:teuthology.orchestra.run.smithi042:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-03-29T02:14:59.473 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:14:59.473 DEBUG:teuthology.orchestra.run.smithi042:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-03-29T02:14:59.525 DEBUG:teuthology.orchestra.run.smithi042:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-03-29T02:14:59.580 INFO:teuthology.orchestra.run.smithi042.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-03-29T02:14:59.953 INFO:tasks.cbt:Installing collectl 2024-03-29T02:14:59.953 DEBUG:teuthology.orchestra.run.smithi042:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-03-29T02:15:00.061 INFO:teuthology.orchestra.run.smithi042.stderr:--2024-03-29 02:15:00-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-03-29T02:15:00.211 INFO:teuthology.orchestra.run.smithi042.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::6812:256f, ... 2024-03-29T02:15:00.221 INFO:teuthology.orchestra.run.smithi042.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected. 2024-03-29T02:15:00.478 INFO:teuthology.orchestra.run.smithi042.stderr:HTTP request sent, awaiting response... 302 Found 2024-03-29T02:15:00.478 INFO:teuthology.orchestra.run.smithi042.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABmBiQkJSrqBBgGmMd8lNYZIgm5Kc3Uppy3zax5WjOvrtaxRhgtGcfXPjWJgfvf4AzTbHFrk7YF9iVkq0KU131A7z04SQ%3D%3D&use_mirror=gigenet&r= [following] 2024-03-29T02:15:00.478 INFO:teuthology.orchestra.run.smithi042.stderr:--2024-03-29 02:15:00-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABmBiQkJSrqBBgGmMd8lNYZIgm5Kc3Uppy3zax5WjOvrtaxRhgtGcfXPjWJgfvf4AzTbHFrk7YF9iVkq0KU131A7z04SQ%3D%3D&use_mirror=gigenet&r= 2024-03-29T02:15:00.478 INFO:teuthology.orchestra.run.smithi042.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-03-29T02:15:00.559 INFO:teuthology.orchestra.run.smithi042.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-03-29T02:15:00.819 INFO:teuthology.orchestra.run.smithi042.stderr:HTTP request sent, awaiting response... 302 Found 2024-03-29T02:15:00.819 INFO:teuthology.orchestra.run.smithi042.stderr:Location: https://gigenet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2024-03-29T02:15:00.819 INFO:teuthology.orchestra.run.smithi042.stderr:--2024-03-29 02:15:00-- https://gigenet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2024-03-29T02:15:00.819 INFO:teuthology.orchestra.run.smithi042.stderr:Resolving gigenet.dl.sourceforge.net (gigenet.dl.sourceforge.net)... 69.65.16.142 2024-03-29T02:15:00.850 INFO:teuthology.orchestra.run.smithi042.stderr:Connecting to gigenet.dl.sourceforge.net (gigenet.dl.sourceforge.net)|69.65.16.142|:443... connected. 2024-03-29T02:15:01.023 INFO:teuthology.orchestra.run.smithi042.stderr:HTTP request sent, awaiting response... 200 OK 2024-03-29T02:15:01.023 INFO:teuthology.orchestra.run.smithi042.stderr:Length: 620693 (606K) [application/x-gzip] 2024-03-29T02:15:01.023 INFO:teuthology.orchestra.run.smithi042.stderr:Saving to: ‘collectl.tar.gz’ 2024-03-29T02:15:01.023 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-29T02:15:01.053 INFO:teuthology.orchestra.run.smithi042.stderr: 0K .......... .......... .......... .......... .......... 8% 1.55M 0s 2024-03-29T02:15:01.094 INFO:teuthology.orchestra.run.smithi042.stderr: 50K .......... .......... .......... .......... .......... 16% 1.22M 0s 2024-03-29T02:15:01.126 INFO:teuthology.orchestra.run.smithi042.stderr: 100K .......... .......... .......... .......... .......... 24% 1.64M 0s 2024-03-29T02:15:01.126 INFO:teuthology.orchestra.run.smithi042.stderr: 150K .......... .......... .......... .......... .......... 32% 58.6M 0s 2024-03-29T02:15:01.126 INFO:teuthology.orchestra.run.smithi042.stderr: 200K .......... .......... .......... .......... .......... 41% 66.3M 0s 2024-03-29T02:15:01.126 INFO:teuthology.orchestra.run.smithi042.stderr: 250K .......... .......... .......... .......... .......... 49% 77.9M 0s 2024-03-29T02:15:01.155 INFO:teuthology.orchestra.run.smithi042.stderr: 300K .......... .......... .......... .......... .......... 57% 1.68M 0s 2024-03-29T02:15:01.158 INFO:teuthology.orchestra.run.smithi042.stderr: 350K .......... .......... .......... .......... .......... 65% 32.8M 0s 2024-03-29T02:15:01.159 INFO:teuthology.orchestra.run.smithi042.stderr: 400K .......... .......... .......... .......... .......... 74% 102M 0s 2024-03-29T02:15:01.159 INFO:teuthology.orchestra.run.smithi042.stderr: 450K .......... .......... .......... .......... .......... 82% 104M 0s 2024-03-29T02:15:01.159 INFO:teuthology.orchestra.run.smithi042.stderr: 500K .......... .......... .......... .......... .......... 90% 41.2M 0s 2024-03-29T02:15:01.160 INFO:teuthology.orchestra.run.smithi042.stderr: 550K .......... .......... .......... .......... .......... 98% 42.6M 0s 2024-03-29T02:15:01.160 INFO:teuthology.orchestra.run.smithi042.stderr: 600K ...... 100% 11.4T=0.1s 2024-03-29T02:15:01.160 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-29T02:15:01.160 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29 02:15:01 (4.31 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-03-29T02:15:01.160 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-29T02:15:01.202 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/ 2024-03-29T02:15:01.203 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/collectl.conf 2024-03-29T02:15:01.203 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/ 2024-03-29T02:15:01.203 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/SlowProc.html 2024-03-29T02:15:01.203 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-03-29T02:15:01.203 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/WhySummary.html 2024-03-29T02:15:01.203 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Socket.html 2024-03-29T02:15:01.203 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/colmux.html 2024-03-29T02:15:01.203 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Tutorial.html 2024-03-29T02:15:01.203 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Gexpr.html 2024-03-29T02:15:01.203 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Import.html 2024-03-29T02:15:01.203 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/style.css 2024-03-29T02:15:01.203 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-03-29T02:15:01.203 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-03-29T02:15:01.204 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/FileNaming.html 2024-03-29T02:15:01.204 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/TheMath.html 2024-03-29T02:15:01.204 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Environmental.html 2024-03-29T02:15:01.204 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Process.html 2024-03-29T02:15:01.204 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-03-29T02:15:01.204 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/HiResTime.html 2024-03-29T02:15:01.204 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-03-29T02:15:01.204 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/slub.jpg 2024-03-29T02:15:01.204 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Lustre.html 2024-03-29T02:15:01.204 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Documentation.html 2024-03-29T02:15:01.204 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Inodes.html 2024-03-29T02:15:01.204 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Graphite.html 2024-03-29T02:15:01.204 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/InputFiles.html 2024-03-29T02:15:01.204 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Export.html 2024-03-29T02:15:01.205 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Interrupts.html 2024-03-29T02:15:01.205 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Startup.html 2024-03-29T02:15:01.205 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Performance.html 2024-03-29T02:15:01.205 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Messages.html 2024-03-29T02:15:01.205 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Logging.html 2024-03-29T02:15:01.205 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Data-detail.html 2024-03-29T02:15:01.205 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/CPUs.html 2024-03-29T02:15:01.205 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Matrix.html 2024-03-29T02:15:01.205 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Data-brief.html 2024-03-29T02:15:01.205 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-03-29T02:15:01.205 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Misc.html 2024-03-29T02:15:01.205 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Examples.html 2024-03-29T02:15:01.205 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/OpenStack.html 2024-03-29T02:15:01.205 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-03-29T02:15:01.205 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-03-29T02:15:01.206 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-03-29T02:15:01.206 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-03-29T02:15:01.206 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Data.html 2024-03-29T02:15:01.206 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Exceptions.html 2024-03-29T02:15:01.206 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-03-29T02:15:01.206 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-03-29T02:15:01.206 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-03-29T02:15:01.206 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-03-29T02:15:01.206 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Playback.html 2024-03-29T02:15:01.206 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Hello.html 2024-03-29T02:15:01.206 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Sockets.html 2024-03-29T02:15:01.206 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Network.html 2024-03-29T02:15:01.206 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Infiniband.html 2024-03-29T02:15:01.207 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Memory.html 2024-03-29T02:15:01.207 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-03-29T02:15:01.207 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/Disks.html 2024-03-29T02:15:01.207 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-03-29T02:15:01.207 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/service/ 2024-03-29T02:15:01.207 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/service/collectl.service 2024-03-29T02:15:01.207 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/initd/ 2024-03-29T02:15:01.207 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/initd/collectl-suse 2024-03-29T02:15:01.207 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/initd/collectl-debian 2024-03-29T02:15:01.207 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/initd/collectl-generic 2024-03-29T02:15:01.207 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/initd/collectl 2024-03-29T02:15:01.207 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/misc.ph 2024-03-29T02:15:01.207 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/gexpr.ph 2024-03-29T02:15:01.208 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/statsd.ph 2024-03-29T02:15:01.208 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/envrules.std 2024-03-29T02:15:01.208 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/hello.ph 2024-03-29T02:15:01.208 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/vmsum.ph 2024-03-29T02:15:01.208 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/proctree.ph 2024-03-29T02:15:01.208 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/README-WINDOWS 2024-03-29T02:15:01.208 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/formatit.ph 2024-03-29T02:15:01.208 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/colmux 2024-03-29T02:15:01.209 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/INSTALL 2024-03-29T02:15:01.209 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/graphite.ph 2024-03-29T02:15:01.209 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/RELEASE-collectl 2024-03-29T02:15:01.209 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/vmstat.ph 2024-03-29T02:15:01.209 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/README 2024-03-29T02:15:01.209 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/man1/ 2024-03-29T02:15:01.209 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/man1/collectl.1 2024-03-29T02:15:01.209 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/man1/colmux.1 2024-03-29T02:15:01.209 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/lexpr.ph 2024-03-29T02:15:01.209 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/client.pl 2024-03-29T02:15:01.209 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/collectl 2024-03-29T02:15:01.210 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/COPYING 2024-03-29T02:15:01.210 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/vnet.ph 2024-03-29T02:15:01.210 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/GPL 2024-03-29T02:15:01.210 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/UNINSTALL 2024-03-29T02:15:01.211 INFO:teuthology.orchestra.run.smithi042.stdout:collectl-4.3.1/ARTISTIC 2024-03-29T02:15:01.258 INFO:teuthology.orchestra.run.smithi042.stderr:./INSTALL: line 90: chkconfig: command not found 2024-03-29T02:15:01.260 DEBUG:teuthology.orchestra.run.smithi042:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-03-29T02:15:01.837 INFO:teuthology.orchestra.run.smithi042.stdout:Last metadata expiration check: 0:03:32 ago on Fri 29 Mar 2024 02:11:29 AM UTC. 2024-03-29T02:15:01.940 INFO:teuthology.orchestra.run.smithi042.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-03-29T02:15:01.943 INFO:teuthology.orchestra.run.smithi042.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-03-29T02:15:02.003 INFO:teuthology.orchestra.run.smithi042.stdout:Dependencies resolved. 2024-03-29T02:15:02.004 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:15:02.004 INFO:teuthology.orchestra.run.smithi042.stdout: Package Arch Version Repository Size 2024-03-29T02:15:02.004 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:15:02.004 INFO:teuthology.orchestra.run.smithi042.stdout:Installing: 2024-03-29T02:15:02.004 INFO:teuthology.orchestra.run.smithi042.stdout: librbd-devel x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 23 k 2024-03-29T02:15:02.004 INFO:teuthology.orchestra.run.smithi042.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-03-29T02:15:02.004 INFO:teuthology.orchestra.run.smithi042.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-03-29T02:15:02.004 INFO:teuthology.orchestra.run.smithi042.stdout: perf x86_64 5.14.0-434.el9 CentOS-AppStream 2.2 M 2024-03-29T02:15:02.004 INFO:teuthology.orchestra.run.smithi042.stdout:Installing dependencies: 2024-03-29T02:15:02.005 INFO:teuthology.orchestra.run.smithi042.stdout: libradospp-devel x86_64 2:19.0.0-2473.g785fc5e8.el9 ceph 30 k 2024-03-29T02:15:02.005 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:15:02.005 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction Summary 2024-03-29T02:15:02.005 INFO:teuthology.orchestra.run.smithi042.stdout:================================================================================ 2024-03-29T02:15:02.005 INFO:teuthology.orchestra.run.smithi042.stdout:Install 5 Packages 2024-03-29T02:15:02.005 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:15:02.005 INFO:teuthology.orchestra.run.smithi042.stdout:Total download size: 2.3 M 2024-03-29T02:15:02.006 INFO:teuthology.orchestra.run.smithi042.stdout:Installed size: 11 M 2024-03-29T02:15:02.006 INFO:teuthology.orchestra.run.smithi042.stdout:Downloading Packages: 2024-03-29T02:15:02.479 INFO:teuthology.orchestra.run.smithi042.stdout:(1/5): perf-5.14.0-434.el9.x86_64.rpm 9.0 MB/s | 2.2 MB 00:00 2024-03-29T02:15:02.496 INFO:teuthology.orchestra.run.smithi042.stdout:(2/5): libradospp-devel-19.0.0-2473.g785fc5e8.e 118 kB/s | 30 kB 00:00 2024-03-29T02:15:02.513 INFO:teuthology.orchestra.run.smithi042.stdout:(3/5): librbd-devel-19.0.0-2473.g785fc5e8.el9.x 84 kB/s | 23 kB 00:00 2024-03-29T02:15:02.714 INFO:teuthology.orchestra.run.smithi042.stdout:(4/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 65 kB/s | 13 kB 00:00 2024-03-29T02:15:02.839 INFO:teuthology.orchestra.run.smithi042.stdout:(5/5): pdsh-2.34-7.el9.x86_64.rpm 330 kB/s | 118 kB 00:00 2024-03-29T02:15:02.842 INFO:teuthology.orchestra.run.smithi042.stdout:-------------------------------------------------------------------------------- 2024-03-29T02:15:02.842 INFO:teuthology.orchestra.run.smithi042.stdout:Total 2.8 MB/s | 2.3 MB 00:00 2024-03-29T02:15:02.842 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction check 2024-03-29T02:15:02.863 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction check succeeded. 2024-03-29T02:15:02.863 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction test 2024-03-29T02:15:02.974 INFO:teuthology.orchestra.run.smithi042.stdout:Transaction test succeeded. 2024-03-29T02:15:02.975 INFO:teuthology.orchestra.run.smithi042.stdout:Running transaction 2024-03-29T02:15:03.271 INFO:teuthology.orchestra.run.smithi042.stdout: Preparing : 1/1 2024-03-29T02:15:03.333 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-03-29T02:15:03.363 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-03-29T02:15:03.394 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : libradospp-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 3/5 2024-03-29T02:15:03.719 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : librbd-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 4/5 2024-03-29T02:15:03.803 INFO:teuthology.orchestra.run.smithi042.stdout: Installing : perf-5.14.0-434.el9.x86_64 5/5 2024-03-29T02:15:04.781 INFO:teuthology.orchestra.run.smithi042.stdout: Running scriptlet: perf-5.14.0-434.el9.x86_64 5/5 2024-03-29T02:15:04.781 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : libradospp-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 1/5 2024-03-29T02:15:04.781 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : librbd-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 2/5 2024-03-29T02:15:04.781 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : perf-5.14.0-434.el9.x86_64 3/5 2024-03-29T02:15:04.782 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-03-29T02:15:05.099 INFO:teuthology.orchestra.run.smithi042.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-03-29T02:15:05.099 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:15:05.099 INFO:teuthology.orchestra.run.smithi042.stdout:Installed: 2024-03-29T02:15:05.099 INFO:teuthology.orchestra.run.smithi042.stdout: libradospp-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:15:05.099 INFO:teuthology.orchestra.run.smithi042.stdout: librbd-devel-2:19.0.0-2473.g785fc5e8.el9.x86_64 2024-03-29T02:15:05.099 INFO:teuthology.orchestra.run.smithi042.stdout: pdsh-2.34-7.el9.x86_64 2024-03-29T02:15:05.100 INFO:teuthology.orchestra.run.smithi042.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-03-29T02:15:05.100 INFO:teuthology.orchestra.run.smithi042.stdout: perf-5.14.0-434.el9.x86_64 2024-03-29T02:15:05.100 INFO:teuthology.orchestra.run.smithi042.stdout: 2024-03-29T02:15:05.100 INFO:teuthology.orchestra.run.smithi042.stdout:Complete! 2024-03-29T02:15:05.451 INFO:tasks.cbt:benchmark: radosbench 2024-03-29T02:15:05.451 DEBUG:teuthology.orchestra.run.smithi042:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-03-29T02:15:05.479 DEBUG:teuthology.orchestra.run.smithi042:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-03-29T02:15:05.854 INFO:teuthology.orchestra.run.smithi042.stderr:02:15:05 - DEBUG - cbt - Settings.cluster: 2024-03-29T02:15:05.854 INFO:teuthology.orchestra.run.smithi042.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-03-29T02:15:05.854 INFO:teuthology.orchestra.run.smithi042.stderr: 'clients': ['ubuntu@smithi042.front.sepia.ceph.com'], 2024-03-29T02:15:05.854 INFO:teuthology.orchestra.run.smithi042.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-03-29T02:15:05.854 INFO:teuthology.orchestra.run.smithi042.stderr: 'head': 'ubuntu@smithi042.front.sepia.ceph.com', 2024-03-29T02:15:05.854 INFO:teuthology.orchestra.run.smithi042.stderr: 'is_teuthology': True, 2024-03-29T02:15:05.854 INFO:teuthology.orchestra.run.smithi042.stderr: 'iterations': 1, 2024-03-29T02:15:05.854 INFO:teuthology.orchestra.run.smithi042.stderr: 'mons': ['ubuntu@smithi042.front.sepia.ceph.com'], 2024-03-29T02:15:05.854 INFO:teuthology.orchestra.run.smithi042.stderr: 'osds': ['ubuntu@smithi042.front.sepia.ceph.com'], 2024-03-29T02:15:05.854 INFO:teuthology.orchestra.run.smithi042.stderr: 'osds_per_node': 3, 2024-03-29T02:15:05.855 INFO:teuthology.orchestra.run.smithi042.stderr: 'pid_dir': '/var/run/ceph', 2024-03-29T02:15:05.855 INFO:teuthology.orchestra.run.smithi042.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2024-03-29T02:15:05.855 INFO:teuthology.orchestra.run.smithi042.stderr: 'pgp_size': 256, 2024-03-29T02:15:05.855 INFO:teuthology.orchestra.run.smithi042.stderr: 'replication': 'replicated'}}, 2024-03-29T02:15:05.855 INFO:teuthology.orchestra.run.smithi042.stderr: 'rebuild_every_test': False, 2024-03-29T02:15:05.855 INFO:teuthology.orchestra.run.smithi042.stderr: 'rgws': {'ubuntu@smithi042.front.sepia.ceph.com': None}, 2024-03-29T02:15:05.855 INFO:teuthology.orchestra.run.smithi042.stderr: 'tmp_dir': '/tmp/cbt', 2024-03-29T02:15:05.855 INFO:teuthology.orchestra.run.smithi042.stderr: 'use_existing': True, 2024-03-29T02:15:05.855 INFO:teuthology.orchestra.run.smithi042.stderr: 'user': 'ubuntu'} 2024-03-29T02:15:05.874 INFO:teuthology.orchestra.run.smithi042.stderr:02:15:05 - INFO - cbt - Pausing for 60s for idle monitoring. 2024-03-29T02:15:05.876 INFO:teuthology.orchestra.run.smithi042.stderr:02:15:05 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:15:05.876 INFO:teuthology.orchestra.run.smithi042.stderr:02:15:05 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:15:05.876 INFO:teuthology.orchestra.run.smithi042.stderr:02:15:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl 2024-03-29T02:15:05.879 INFO:teuthology.orchestra.run.smithi042.stderr:02:15:05 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:15:05.880 INFO:teuthology.orchestra.run.smithi042.stderr:02:15:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-03-29T02:15:05.881 INFO:teuthology.orchestra.run.smithi042.stderr:02:15:05 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:15:05.881 INFO:teuthology.orchestra.run.smithi042.stderr:02:15:05 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:15:05.881 INFO:teuthology.orchestra.run.smithi042.stderr:02:15:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf 2024-03-29T02:15:05.884 INFO:teuthology.orchestra.run.smithi042.stderr:02:15:05 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:15:05.884 INFO:teuthology.orchestra.run.smithi042.stderr:02:15:05 - DEBUG - cbt - PerfMonitoring: in local_node 2024-03-29T02:15:05.885 INFO:teuthology.orchestra.run.smithi042.stderr:02:15:05 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-03-29T02:16:05.943 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:05 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:05.944 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-03-29T02:16:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:05 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:05.987 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-03-29T02:16:06.031 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:06.031 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2024-03-29T02:16:06.033 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:06.033 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2024-03-29T02:16:06.034 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:16:06.035 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:06.035 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/* 2024-03-29T02:16:06.052 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:06.052 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/$(basename ${i}).smithi042.front.sepia.ceph.com ; done 2024-03-29T02:16:06.056 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:16:06.056 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:06.056 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo killall -9 rados 2024-03-29T02:16:06.078 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - WARNING - cbt - sudo killall -9 rados 2024-03-29T02:16:06.078 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - WARNING - cbt - error 1 seen, continuing anyway... 2024-03-29T02:16:06.097 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3. 2024-03-29T02:16:06.097 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004 2024-03-29T02:16:06.097 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:16:06.098 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:06.098 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004 2024-03-29T02:16:06.114 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:16:06.115 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:06.115 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl 2024-03-29T02:16:06.117 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:06.117 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-03-29T02:16:06.119 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:16:06.119 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:06.119 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf 2024-03-29T02:16:06.122 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:06.122 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - PerfMonitoring: in local_node 2024-03-29T02:16:06.122 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-03-29T02:16:06.122 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:16:06.122 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:06.122 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2024-03-29T02:16:06.441 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:16:06.441 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:06.441 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:06 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2024-03-29T02:16:07.389 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:07 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:16:07.389 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:07 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:07.389 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:07 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2024-03-29T02:16:08.395 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:08 - INFO - cbt - Checking Health after pool creation. 2024-03-29T02:16:13.400 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:13 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:16:13.401 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:13 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:13.401 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-03-29T02:16:13.741 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:13 - INFO - cbt - Final Pool Health Check. 2024-03-29T02:16:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:18 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:16:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:18 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:18.747 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:18 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-03-29T02:16:19.085 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:19 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:19.085 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-03-29T02:16:19.108 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:19 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:19.108 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-03-29T02:16:19.144 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:19 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:19.144 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2024-03-29T02:16:19.146 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:19 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:19.146 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2024-03-29T02:16:19.147 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:19 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:16:19.147 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:19 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:19.148 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-03-29T02:16:19.224 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:19 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:19.224 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-03-29T02:16:19.800 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:19 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:16:19.800 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:19 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:19.800 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:19 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2024-03-29T02:16:20.173 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:20 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:16:20.173 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:20 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:20.173 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:20 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write 2024-03-29T02:16:20.240 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:20 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:16:20.240 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:20 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:20.241 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:20 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/ceph_settings.out 2024-03-29T02:16:21.031 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:21 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:16:21.031 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:21 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:21.031 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/collectl 2024-03-29T02:16:21.034 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:21 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:21.034 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-03-29T02:16:21.035 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:21 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:16:21.035 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:21 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:21.035 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf 2024-03-29T02:16:21.038 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:21 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:21.038 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:21 - DEBUG - cbt - PerfMonitoring: in local_node 2024-03-29T02:16:21.038 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:21 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-03-29T02:16:21.039 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:21 - INFO - cbt - Running radosbench write test. 2024-03-29T02:16:21.039 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:21 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:16:21.039 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:21 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:21.039 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/output.0 2024-03-29T02:16:21.040 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:21 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:16:21.040 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:21 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:16:21.040 INFO:teuthology.orchestra.run.smithi042.stderr:02:16:21 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/output.1 2024-03-29T02:17:22.198 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:22 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:17:22.198 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-03-29T02:17:22.234 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:22 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:17:22.235 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-03-29T02:17:22.274 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:22 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:17:22.274 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf/perf.data 2024-03-29T02:17:22.276 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:22 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:17:22.276 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf/perf_stat.* 2024-03-29T02:17:22.278 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:22 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:17:22.278 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:22 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:17:22.278 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/historic_ops.out 2024-03-29T02:17:22.595 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:22 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:17:22.602 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:22 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:17:22.602 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:22 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/* 2024-03-29T02:17:22.619 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:22 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:17:22.619 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:22 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/$(basename ${i}).smithi042.front.sepia.ceph.com ; done 2024-03-29T02:17:22.638 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:22 - INFO - cbt - Convert results to json format. 2024-03-29T02:17:22.638 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:22 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:17:22.639 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:22 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:17:22.639 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:22 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:17:22.639 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-03-29T02:17:22.938 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:22 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:17:22.938 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:22 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-03-29T02:17:23.241 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:23 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:17:23.242 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:23 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:17:23.242 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2024-03-29T02:17:23.615 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:23 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:17:23.615 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:23 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:17:23.615 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:23 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand 2024-03-29T02:17:23.669 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:23 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:17:23.669 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:23 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:17:23.669 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:23 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/ceph_settings.out 2024-03-29T02:17:24.501 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:24 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:17:24.501 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:24 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:17:24.501 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/collectl 2024-03-29T02:17:24.504 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:24 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:17:24.504 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-03-29T02:17:24.505 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:24 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:17:24.506 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:24 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:17:24.506 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/perf 2024-03-29T02:17:24.509 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:24 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:17:24.509 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:24 - DEBUG - cbt - PerfMonitoring: in local_node 2024-03-29T02:17:24.509 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:24 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-03-29T02:17:24.509 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:24 - INFO - cbt - Running radosbench rand test. 2024-03-29T02:17:24.509 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:24 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:17:24.509 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:24 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:17:24.509 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 rand --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/output.0 2024-03-29T02:17:24.510 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:24 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:17:24.510 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:24 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:17:24.511 INFO:teuthology.orchestra.run.smithi042.stderr:02:17:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 rand --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/output.1 2024-03-29T02:18:24.675 INFO:teuthology.orchestra.run.smithi042.stderr:02:18:24 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:18:24.676 INFO:teuthology.orchestra.run.smithi042.stderr:02:18:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-03-29T02:18:24.709 INFO:teuthology.orchestra.run.smithi042.stderr:02:18:24 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:18:24.710 INFO:teuthology.orchestra.run.smithi042.stderr:02:18:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-03-29T02:18:24.744 INFO:teuthology.orchestra.run.smithi042.stderr:02:18:24 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:18:24.744 INFO:teuthology.orchestra.run.smithi042.stderr:02:18:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/perf/perf.data 2024-03-29T02:18:24.746 INFO:teuthology.orchestra.run.smithi042.stderr:02:18:24 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:18:24.746 INFO:teuthology.orchestra.run.smithi042.stderr:02:18:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/perf/perf_stat.* 2024-03-29T02:18:24.747 INFO:teuthology.orchestra.run.smithi042.stderr:02:18:24 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:18:24.747 INFO:teuthology.orchestra.run.smithi042.stderr:02:18:24 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:18:24.747 INFO:teuthology.orchestra.run.smithi042.stderr:02:18:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/historic_ops.out 2024-03-29T02:18:25.076 INFO:teuthology.orchestra.run.smithi042.stderr:02:18:25 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:18:25.094 INFO:teuthology.orchestra.run.smithi042.stderr:02:18:25 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:18:25.094 INFO:teuthology.orchestra.run.smithi042.stderr:02:18:25 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/* 2024-03-29T02:18:25.111 INFO:teuthology.orchestra.run.smithi042.stderr:02:18:25 - DEBUG - cbt - get_fqdn_local()=smithi042.front.sepia.ceph.com 2024-03-29T02:18:25.111 INFO:teuthology.orchestra.run.smithi042.stderr:02:18:25 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/rand/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/$(basename ${i}).smithi042.front.sepia.ceph.com ; done 2024-03-29T02:18:25.188 INFO:teuthology.orchestra.run.smithi042.stderr:02:18:25 - INFO - cbt - Convert results to json format. 2024-03-29T02:18:25.189 INFO:teuthology.orchestra.run.smithi042.stderr:02:18:25 - DEBUG - cbt - Nodes : ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:18:25.213 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2024-03-29T02:18:25.223 DEBUG:teuthology.orchestra.run.smithi042:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2024-03-29T02:18:25.436 INFO:tasks.cbt_performance:Collecting CBT performance data config 2024-03-29T02:18:25.436 INFO:tasks.cbt_performance:reading cbt preferences 2024-03-29T02:18:25.436 DEBUG:teuthology.orchestra.run.smithi042:> find /home/ubuntu/cephtest -name 'json_output.*' 2024-03-29T02:18:25.502 INFO:teuthology.orchestra.run.smithi042.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/json_output.1.smithi042.front.sepia.ceph.com 2024-03-29T02:18:25.502 INFO:teuthology.orchestra.run.smithi042.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/json_output.0.smithi042.front.sepia.ceph.com 2024-03-29T02:18:25.502 INFO:teuthology.orchestra.run.smithi042.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/json_output.1.smithi042.front.sepia.ceph.com 2024-03-29T02:18:25.503 INFO:teuthology.orchestra.run.smithi042.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/json_output.0.smithi042.front.sepia.ceph.com 2024-03-29T02:18:25.503 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:18:25.504 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/json_output.1.smithi042.front.sepia.ceph.com of=/dev/stdout 2024-03-29T02:18:25.528 INFO:tasks.cbt_performance:reading total cpu cycles 2024-03-29T02:18:25.528 DEBUG:teuthology.orchestra.run.smithi042:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write -name 'perf_stat.*' 2024-03-29T02:18:25.584 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-03-29T02:18:25.584 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-03-29T02:18:25.584 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:18:25.584 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write/json_output.0.smithi042.front.sepia.ceph.com of=/dev/stdout 2024-03-29T02:18:25.645 INFO:tasks.cbt_performance:reading total cpu cycles 2024-03-29T02:18:25.645 DEBUG:teuthology.orchestra.run.smithi042:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/write -name 'perf_stat.*' 2024-03-29T02:18:25.700 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-03-29T02:18:25.700 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-03-29T02:18:25.700 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:18:25.700 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/json_output.1.smithi042.front.sepia.ceph.com of=/dev/stdout 2024-03-29T02:18:25.755 INFO:tasks.cbt_performance:reading total cpu cycles 2024-03-29T02:18:25.756 DEBUG:teuthology.orchestra.run.smithi042:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand -name 'perf_stat.*' 2024-03-29T02:18:25.812 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-03-29T02:18:25.812 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-03-29T02:18:25.812 DEBUG:teuthology.orchestra.run.smithi042:> set -ex 2024-03-29T02:18:25.813 DEBUG:teuthology.orchestra.run.smithi042:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand/json_output.0.smithi042.front.sepia.ceph.com of=/dev/stdout 2024-03-29T02:18:25.869 INFO:tasks.cbt_performance:reading total cpu cycles 2024-03-29T02:18:25.869 DEBUG:teuthology.orchestra.run.smithi042:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-a0b9bdc3/rand -name 'perf_stat.*' 2024-03-29T02:18:25.925 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-03-29T02:18:25.925 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-03-29T02:18:25.935 INFO:tasks.cbt_performance:Data inserted successfully. 2024-03-29T02:18:25.941 INFO:tasks.cbt_performance:Data inserted successfully. 2024-03-29T02:18:25.947 INFO:tasks.cbt_performance:Data inserted successfully. 2024-03-29T02:18:25.953 INFO:tasks.cbt_performance:Data inserted successfully. 2024-03-29T02:18:25.953 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T02:18:25.963 DEBUG:teuthology.orchestra.run.smithi042:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-29T02:18:26.098 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:18:26.092+0000 7f521b99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:18:26.100 INFO:teuthology.orchestra.run.smithi042.stderr:2024-03-29T02:18:26.101+0000 7f521b99d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-03-29T02:18:26.320 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-29T02:18:26.321 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T02:18:26.321 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T02:18:26.321 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T02:18:26.412 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T02:18:26.412 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T02:18:26.412 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T02:18:26.501 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T02:18:26.502 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T02:18:26.502 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T02:18:26.595 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T02:18:26.595 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T02:18:26.595 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T02:18:26.595 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T02:18:26.621 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T02:18:26.621 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T02:18:26.622 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T02:18:26.622 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T02:18:26.673 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T02:18:26.673 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T02:18:26.674 DEBUG:teuthology.orchestra.run.smithi042:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v 'overall HEALTH' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-29T02:18:26.796 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:18:26.796 DEBUG:teuthology.orchestra.run.smithi042:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T02:18:27.002 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:18:27.002 DEBUG:teuthology.orchestra.run.smithi042:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T02:18:27.089 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:18:27.089 DEBUG:teuthology.orchestra.run.smithi042:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T02:18:27.147 INFO:tasks.ceph:Archiving mon data... 2024-03-29T02:18:27.148 DEBUG:teuthology.misc:Transferring archived files from smithi042:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629056/data/mon.a.tgz 2024-03-29T02:18:27.148 DEBUG:teuthology.orchestra.run.smithi042:> mktemp 2024-03-29T02:18:27.163 INFO:teuthology.orchestra.run.smithi042.stdout:/tmp/tmp.VL2SlwlPPe 2024-03-29T02:18:27.163 DEBUG:teuthology.orchestra.run.smithi042:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.VL2SlwlPPe 2024-03-29T02:18:27.341 DEBUG:teuthology.orchestra.run.smithi042:> sudo chmod 0666 /tmp/tmp.VL2SlwlPPe 2024-03-29T02:18:27.425 DEBUG:teuthology.orchestra.remote:smithi042:/tmp/tmp.VL2SlwlPPe is 190KB 2024-03-29T02:18:27.477 DEBUG:teuthology.orchestra.run.smithi042:> rm -fr /tmp/tmp.VL2SlwlPPe 2024-03-29T02:18:27.491 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T02:18:27.491 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T02:18:27.616 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T02:18:27.616 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T02:18:27.618 DEBUG:teuthology.misc:Transferring archived files from smithi042:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629056/remote/smithi042/crash 2024-03-29T02:18:27.619 DEBUG:teuthology.orchestra.run.smithi042:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T02:18:27.681 INFO:tasks.ceph:Compressing logs... 2024-03-29T02:18:27.681 DEBUG:teuthology.orchestra.run.smithi042:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T02:18:27.782 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31135.log 2024-03-29T02:18:27.782 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T02:18:27.782 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30867.log 2024-03-29T02:18:27.782 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31894.log 2024-03-29T02:18:27.782 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31513.log 2024-03-29T02:18:27.782 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33204.log 2024-03-29T02:18:27.782 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31600.log 2024-03-29T02:18:27.782 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32983.log 2024-03-29T02:18:27.782 INFO:teuthology.orchestra.run.smithi042.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.33204.log.gz 2024-03-29T02:18:27.783 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29938.log/var/log/ceph/ceph-client.admin.31600.log: 2024-03-29T02:18:27.783 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31766.log 2024-03-29T02:18:27.783 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.29938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31765.log 2024-03-29T02:18:27.783 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31639.log 2024-03-29T02:18:27.783 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33047.log 2024-03-29T02:18:27.783 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31640.log 2024-03-29T02:18:27.783 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31639.log: /var/log/ceph/ceph-client.admin.33047.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T02:18:27.783 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31054.log 2024-03-29T02:18:27.783 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30970.log 2024-03-29T02:18:27.783 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30404.log 2024-03-29T02:18:27.783 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30216.log 2024-03-29T02:18:27.783 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30404.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T02:18:27.783 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31514.log 2024-03-29T02:18:27.783 INFO:teuthology.orchestra.run.smithi042.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30026.log 2024-03-29T02:18:27.784 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.31514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30909.log 2024-03-29T02:18:27.784 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31726.log 2024-03-29T02:18:27.784 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32934.log 2024-03-29T02:18:27.784 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30136.log 2024-03-29T02:18:27.784 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30811.log 2024-03-29T02:18:27.784 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31262.log 2024-03-29T02:18:27.784 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31852.log 2024-03-29T02:18:27.784 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31387.log 2024-03-29T02:18:27.784 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31852.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28567.log 2024-03-29T02:18:27.784 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31936.log 2024-03-29T02:18:27.784 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph.tmp-client.admin.28567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30176.log 2024-03-29T02:18:27.784 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31388.log 2024-03-29T02:18:27.784 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30176.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T02:18:27.784 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33203.log 2024-03-29T02:18:27.784 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32897.log 2024-03-29T02:18:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32958.log 2024-03-29T02:18:27.785 INFO:teuthology.orchestra.run.smithi042.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.33203.log.gz 2024-03-29T02:18:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31261.log 2024-03-29T02:18:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32958.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T02:18:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31261.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T02:18:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31134.log 2024-03-29T02:18:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T02:18:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31136.log 2024-03-29T02:18:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30374.log 2024-03-29T02:18:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33048.log 2024-03-29T02:18:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.30374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32832.log 2024-03-29T02:18:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31474.log 2024-03-29T02:18:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.32832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31348.log 2024-03-29T02:18:27.785 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31012.log 2024-03-29T02:18:27.786 INFO:teuthology.orchestra.run.smithi042.stderr:gzip/var/log/ceph/ceph-client.admin.31348.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31222.log 2024-03-29T02:18:27.786 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33504.log 2024-03-29T02:18:27.786 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.31222.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31135.log.gz 2024-03-29T02:18:27.786 INFO:teuthology.orchestra.run.smithi042.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31262.log.gz 2024-03-29T02:18:27.786 INFO:teuthology.orchestra.run.smithi042.stderr:/var/log/ceph/ceph-client.admin.33504.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33504.log.gz 2024-03-29T02:18:27.786 INFO:teuthology.orchestra.run.smithi042.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31513.log.gz 2024-03-29T02:18:27.786 INFO:teuthology.orchestra.run.smithi042.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31600.log.gz 50.0% 2024-03-29T02:18:27.786 INFO:teuthology.orchestra.run.smithi042.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31766.log.gz 2024-03-29T02:18:27.786 INFO:teuthology.orchestra.run.smithi042.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31726.log.gz 2024-03-29T02:18:27.786 INFO:teuthology.orchestra.run.smithi042.stderr: 50.0% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31639.log.gz 2024-03-29T02:18:27.786 INFO:teuthology.orchestra.run.smithi042.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31514.log.gz 2024-03-29T02:18:27.786 INFO:teuthology.orchestra.run.smithi042.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31640.log.gz 2024-03-29T02:18:27.786 INFO:teuthology.orchestra.run.smithi042.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31894.log.gz 2024-03-29T02:18:27.786 INFO:teuthology.orchestra.run.smithi042.stderr: 49.6% 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31765.log.gz -- replaced with /var/log/ceph/ceph-client.admin.31852.log.gz 2024-03-29T02:18:27.787 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-29T02:18:27.787 INFO:teuthology.orchestra.run.smithi042.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29938.log.gz 2024-03-29T02:18:27.787 INFO:teuthology.orchestra.run.smithi042.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30026.log.gz 2024-03-29T02:18:27.787 INFO:teuthology.orchestra.run.smithi042.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30404.log.gz 2024-03-29T02:18:27.789 INFO:teuthology.orchestra.run.smithi042.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32934.log.gz 2024-03-29T02:18:27.789 INFO:teuthology.orchestra.run.smithi042.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32897.log.gz 2024-03-29T02:18:27.789 INFO:teuthology.orchestra.run.smithi042.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31222.log.gz 2024-03-29T02:18:27.789 INFO:teuthology.orchestra.run.smithi042.stderr: 87.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T02:18:27.789 INFO:teuthology.orchestra.run.smithi042.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32832.log.gz 2024-03-29T02:18:27.789 INFO:teuthology.orchestra.run.smithi042.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.32958.log.gz 2024-03-29T02:18:27.789 INFO:teuthology.orchestra.run.smithi042.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32983.log.gz 2024-03-29T02:18:27.789 INFO:teuthology.orchestra.run.smithi042.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30811.log.gz 2024-03-29T02:18:27.790 INFO:teuthology.orchestra.run.smithi042.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30867.log.gz 2024-03-29T02:18:27.790 INFO:teuthology.orchestra.run.smithi042.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30909.log.gz 2024-03-29T02:18:27.790 INFO:teuthology.orchestra.run.smithi042.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30970.log.gz 2024-03-29T02:18:27.790 INFO:teuthology.orchestra.run.smithi042.stderr: 47.6% -- replaced with /var/log/ceph/ceph-client.admin.31012.log.gz 2024-03-29T02:18:27.790 INFO:teuthology.orchestra.run.smithi042.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31054.log.gz 2024-03-29T02:18:27.790 INFO:teuthology.orchestra.run.smithi042.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31136.log.gz 2024-03-29T02:18:27.790 INFO:teuthology.orchestra.run.smithi042.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31134.log.gz 2024-03-29T02:18:27.790 INFO:teuthology.orchestra.run.smithi042.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31261.log.gz 2024-03-29T02:18:27.790 INFO:teuthology.orchestra.run.smithi042.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31348.log.gz 2024-03-29T02:18:27.790 INFO:teuthology.orchestra.run.smithi042.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31388.log.gz 2024-03-29T02:18:27.790 INFO:teuthology.orchestra.run.smithi042.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.31387.log.gz 2024-03-29T02:18:27.790 INFO:teuthology.orchestra.run.smithi042.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.31474.log.gz 2024-03-29T02:18:27.790 INFO:teuthology.orchestra.run.smithi042.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.31936.log.gz 2024-03-29T02:18:27.791 INFO:teuthology.orchestra.run.smithi042.stderr: 85.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T02:18:27.794 INFO:teuthology.orchestra.run.smithi042.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30136.log.gz 2024-03-29T02:18:27.794 INFO:teuthology.orchestra.run.smithi042.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30176.log.gz 2024-03-29T02:18:27.795 INFO:teuthology.orchestra.run.smithi042.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30216.log.gz 2024-03-29T02:18:27.797 INFO:teuthology.orchestra.run.smithi042.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30374.log.gz 2024-03-29T02:18:27.802 INFO:teuthology.orchestra.run.smithi042.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28567.log.gz 2024-03-29T02:18:27.813 INFO:teuthology.orchestra.run.smithi042.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T02:18:27.885 INFO:teuthology.orchestra.run.smithi042.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T02:18:27.927 INFO:teuthology.orchestra.run.smithi042.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.33047.log.gz 2024-03-29T02:18:27.927 INFO:teuthology.orchestra.run.smithi042.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.33048.log.gz 2024-03-29T02:18:33.639 INFO:teuthology.orchestra.run.smithi042.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T02:18:33.951 INFO:teuthology.orchestra.run.smithi042.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T02:18:33.965 INFO:teuthology.orchestra.run.smithi042.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T02:18:33.967 INFO:teuthology.orchestra.run.smithi042.stderr: 2024-03-29T02:18:33.967 INFO:teuthology.orchestra.run.smithi042.stderr:real 0m6.231s 2024-03-29T02:18:33.968 INFO:teuthology.orchestra.run.smithi042.stderr:user 0m3.961s 2024-03-29T02:18:33.968 INFO:teuthology.orchestra.run.smithi042.stderr:sys 0m0.238s 2024-03-29T02:18:33.968 INFO:tasks.ceph:Archiving logs... 2024-03-29T02:18:33.968 DEBUG:teuthology.misc:Transferring archived files from smithi042:/var/log/ceph to /home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629056/remote/smithi042/log 2024-03-29T02:18:33.969 DEBUG:teuthology.orchestra.run.smithi042:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T02:18:34.704 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T02:18:34.714 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T02:18:34.715 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T02:18:34.777 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T02:18:34.786 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T02:18:34.786 DEBUG:teuthology.orchestra.run.smithi042:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T02:18:34.806 INFO:teuthology.orchestra.run.smithi042.stderr:bash: line 1: ntpq: command not found 2024-03-29T02:18:34.871 INFO:teuthology.orchestra.run.smithi042.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T02:18:34.871 INFO:teuthology.orchestra.run.smithi042.stdout:=============================================================================== 2024-03-29T02:18:34.871 INFO:teuthology.orchestra.run.smithi042.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 8 +772us[ +772us] +/- 74ms 2024-03-29T02:18:34.871 INFO:teuthology.orchestra.run.smithi042.stdout:^* hv02.front.sepia.ceph.com 3 6 377 10 -19us[ -50us] +/- 79ms 2024-03-29T02:18:34.871 INFO:teuthology.orchestra.run.smithi042.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 11 -919us[ -950us] +/- 82ms 2024-03-29T02:18:34.871 INFO:teuthology.orchestra.run.smithi042.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T02:18:34.877 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T02:18:34.885 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T02:18:34.886 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T02:18:34.957 DEBUG:teuthology.orchestra.run.smithi042:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T02:18:35.090 DEBUG:teuthology.orchestra.run.smithi042:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-29T02:18:35.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T02:18:35.121 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T02:18:35.132 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T02:18:35.170 INFO:teuthology.task.internal:Duration was 730.576757 seconds 2024-03-29T02:18:35.170 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T02:18:35.200 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T02:18:35.200 DEBUG:teuthology.orchestra.run.smithi042:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T02:18:35.340 INFO:teuthology.orchestra.run.smithi042.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T02:18:35.779 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T02:18:35.780 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi042.front.sepia.ceph.com 2024-03-29T02:18:35.780 DEBUG:teuthology.orchestra.run.smithi042:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T02:18:35.801 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T02:18:35.801 DEBUG:teuthology.orchestra.run.smithi042:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T02:18:35.874 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T02:18:35.874 DEBUG:teuthology.orchestra.run.smithi042:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T02:18:36.018 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T02:18:36.028 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T02:18:36.028 DEBUG:teuthology.orchestra.run.smithi042:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T02:18:36.061 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T02:18:36.071 DEBUG:teuthology.orchestra.run.smithi042:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T02:18:36.143 INFO:teuthology.orchestra.run.smithi042.stdout:kernel.core_pattern = core 2024-03-29T02:18:36.166 DEBUG:teuthology.orchestra.run.smithi042:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T02:18:36.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T02:18:36.221 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T02:18:36.231 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T02:18:36.232 DEBUG:teuthology.misc:Transferring archived files from smithi042:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yingxin-2024-03-29_01:54:41-crimson-rados-ci-yingxin-crimson-load-obc-3-distro-default-smithi/7629056/remote/smithi042 2024-03-29T02:18:36.232 DEBUG:teuthology.orchestra.run.smithi042:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T02:18:36.362 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T02:18:36.362 DEBUG:teuthology.orchestra.run.smithi042:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T02:18:36.405 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T02:18:36.416 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T02:18:36.417 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T02:18:36.449 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T02:18:36.450 DEBUG:teuthology.orchestra.run.smithi042:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T02:18:36.464 INFO:teuthology.orchestra.run.smithi042.stdout: 266033 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 02:18 /home/ubuntu/cephtest 2024-03-29T02:18:36.465 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T02:18:36.475 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T02:18:36.533 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/Rfy7LkB4k/rados-bench?orgId=1&var-branch_name=ci-yingxin-crimson-load-obc-3&var-job_id_selected=7629056 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_rand_read} duration: 730.5767571926117 flavor: crimson owner: scheduled_yingxin@teuthology success: true 2024-03-29T02:18:36.533 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T02:18:36.614 INFO:teuthology.run:pass