2023-11-29T08:36:16.337 INFO:root:teuthology version: 0.0.1.dev212+g735a43c 2023-11-29T08:36:16.337 DEBUG:teuthology.run:Teuthology command: teuthology --name matan-2023-11-29_08:30:41-crimson-rados-wip-matanb-crimson-snaptrimevent-lifetime-v2-distro-crimson-smithi --verbose --owner scheduled_matan@teuthology --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_write} --archive /home/teuthworker/archive/matan-2023-11-29_08:30:41-crimson-rados-wip-matanb-crimson-snaptrimevent-lifetime-v2-distro-crimson-smithi/7470487 -- /home/teuthworker/archive/matan-2023-11-29_08:30:41-crimson-rados-wip-matanb-crimson-snaptrimevent-lifetime-v2-distro-crimson-smithi/7470487/orig.config.yaml 2023-11-29T08:36:16.376 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-29T08:36:16.449 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2023-11-29_08:30:41-crimson-rados-wip-matanb-crimson-snaptrimevent-lifetime-v2-distro-crimson-smithi/7470487 branch: wip-matanb-crimson-snaptrimevent-lifetime-v2 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_write} email: null first_in_suite: false job_id: '7470487' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2023-11-29_08:30:41-crimson-rados-wip-matanb-crimson-snaptrimevent-lifetime-v2-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-snaptrimevent-lifetime-v2 ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: 412bfade5c44735b71d7c14eb55649269cd32a35 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 412bfade5c44735b71d7c14eb55649269cd32a35 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 3d760f113bd9f3a2e513474d34227633cd06f1ea owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 4297 sha1: 412bfade5c44735b71d7c14eb55649269cd32a35 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_3d760f113bd9f3a2e513474d34227633cd06f1ea/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 3d760f113bd9f3a2e513474d34227633cd06f1ea targets: smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGsnONvKlQj1qDG+OX3AiyN17NuwYgVbVMLBMf+Q/SyQ5q+jg/0ZkroA3iuHPBwo1hTF8rBOTrrWA1rTL3h+JUI= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: librbdfio: iodepth: - 32 log_avg_msec: 100 mode: - randwrite norandommap: true op_size: - 4194304 osd_ra: - 4096 pool_profile: rbd procs_per_volume: - 1 time: 60 vol_size: 4096 volumes_per_client: - 2 cluster: iterations: 1 osds_per_node: 3 pool_profiles: rbd: pg_size: 128 pgp_size: 128 replication: 3 user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 735a43c3d8e74286f980f1d14817fb093cd8147e timestamp: 2023-11-29_08:30:41 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-11-29T08:36:16.449 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_3d760f113bd9f3a2e513474d34227633cd06f1ea/qa; will attempt to use it 2023-11-29T08:36:16.450 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_3d760f113bd9f3a2e513474d34227633cd06f1ea/qa/tasks 2023-11-29T08:36:16.450 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-29T08:36:16.452 INFO:teuthology.task.internal:Checking packages... 2023-11-29T08:36:16.473 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '412bfade5c44735b71d7c14eb55649269cd32a35' 2023-11-29T08:36:16.473 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-29T08:36:16.474 INFO:teuthology.packaging:ref: None 2023-11-29T08:36:16.474 INFO:teuthology.packaging:tag: None 2023-11-29T08:36:16.474 INFO:teuthology.packaging:branch: wip-matanb-crimson-snaptrimevent-lifetime-v2 2023-11-29T08:36:16.474 INFO:teuthology.packaging:sha1: 412bfade5c44735b71d7c14eb55649269cd32a35 2023-11-29T08:36:16.474 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-snaptrimevent-lifetime-v2 2023-11-29T08:36:16.616 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7483.g412bfade 2023-11-29T08:36:16.616 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-29T08:36:16.622 INFO:teuthology.task.internal:no buildpackages task found 2023-11-29T08:36:16.623 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-29T08:36:16.646 INFO:teuthology.task.internal:Saving configuration 2023-11-29T08:36:16.660 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-29T08:36:16.666 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-29T08:36:16.686 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-11-29_08:30:41-crimson-rados-wip-matanb-crimson-snaptrimevent-lifetime-v2-distro-crimson-smithi/7470487', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-29 08:31:47.177063', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGsnONvKlQj1qDG+OX3AiyN17NuwYgVbVMLBMf+Q/SyQ5q+jg/0ZkroA3iuHPBwo1hTF8rBOTrrWA1rTL3h+JUI='} 2023-11-29T08:36:16.687 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-29T08:36:16.692 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-11-29T08:36:16.692 INFO:teuthology.run_tasks:Running task console_log... 2023-11-29T08:36:16.728 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc43d3829d0>, signals=[15]) 2023-11-29T08:36:16.729 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-29T08:36:16.735 INFO:teuthology.task.internal:Opening connections... 2023-11-29T08:36:16.735 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:36:16.736 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-29T08:36:16.835 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-29T08:36:16.841 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2023-11-29T08:36:16.902 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2023-11-29T08:36:16.903 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2023-11-29T08:36:16.959 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="CentOS Stream" 2023-11-29T08:36:16.960 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="9" 2023-11-29T08:36:16.960 INFO:teuthology.orchestra.run.smithi031.stdout:ID="centos" 2023-11-29T08:36:16.960 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="rhel fedora" 2023-11-29T08:36:16.960 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="9" 2023-11-29T08:36:16.960 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el9" 2023-11-29T08:36:16.960 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-29T08:36:16.960 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2023-11-29T08:36:16.960 INFO:teuthology.orchestra.run.smithi031.stdout:LOGO="fedora-logo-icon" 2023-11-29T08:36:16.960 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-29T08:36:16.960 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://centos.org/" 2023-11-29T08:36:16.960 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-29T08:36:16.960 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-29T08:36:16.960 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-29T08:36:16.961 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2023-11-29T08:36:16.984 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-29T08:36:16.993 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-29T08:36:16.999 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-29T08:36:16.999 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2023-11-29T08:36:17.016 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-29T08:36:17.022 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-29T08:36:17.022 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2023-11-29T08:36:17.084 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-29T08:36:17.112 INFO:teuthology.run_tasks:Running task kernel... 2023-11-29T08:36:17.127 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-29T08:36:17.127 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-29T08:36:17.127 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-29T08:36:17.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-29T08:36:17.143 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2023-11-29T08:36:17.199 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-361.el9.x86_64 2023-11-29T08:36:17.199 INFO:teuthology.task.kernel:Running kernel on smithi031: 5.14.0-361.el9.x86_64 2023-11-29T08:36:17.199 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum install -y kernel 2023-11-29T08:36:19.480 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - AppStream 9.5 MB/s | 6.4 MB 00:00 2023-11-29T08:36:21.088 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2023-11-29T08:36:23.062 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 7.9 MB 00:01 2023-11-29T08:36:24.921 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 27 MB/s | 18 MB 00:00 2023-11-29T08:36:29.039 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 11 MB/s | 5.8 MB 00:00 2023-11-29T08:36:30.489 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 15 kB 00:00 2023-11-29T08:36:31.840 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 16 MB/s | 20 MB 00:01 2023-11-29T08:36:36.949 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2023-11-29T08:36:39.313 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-11-29T08:36:39.313 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-11-29T08:36:39.411 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:36:39.421 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:36:39.421 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2023-11-29T08:36:39.422 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:36:39.422 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-11-29T08:36:39.422 INFO:teuthology.orchestra.run.smithi031.stdout: kernel x86_64 5.14.0-388.el9 baseos 5.6 M 2023-11-29T08:36:39.422 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-11-29T08:36:39.422 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-core x86_64 5.14.0-388.el9 baseos 20 M 2023-11-29T08:36:39.422 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules x86_64 5.14.0-388.el9 baseos 38 M 2023-11-29T08:36:39.422 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-core x86_64 5.14.0-388.el9 baseos 32 M 2023-11-29T08:36:39.422 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:36:39.422 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-11-29T08:36:39.422 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:36:39.422 INFO:teuthology.orchestra.run.smithi031.stdout:Install 4 Packages 2023-11-29T08:36:39.422 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:36:39.423 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 96 M 2023-11-29T08:36:39.423 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 121 M 2023-11-29T08:36:39.423 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-11-29T08:36:40.099 INFO:teuthology.orchestra.run.smithi031.stdout:(1/4): kernel-5.14.0-388.el9.x86_64.rpm 10 MB/s | 5.6 MB 00:00 2023-11-29T08:36:40.708 INFO:teuthology.orchestra.run.smithi031.stdout:(2/4): kernel-core-5.14.0-388.el9.x86_64.rpm 18 MB/s | 20 MB 00:01 2023-11-29T08:36:41.325 INFO:teuthology.orchestra.run.smithi031.stdout:(3/4): kernel-modules-5.14.0-388.el9.x86_64.rpm 21 MB/s | 38 MB 00:01 2023-11-29T08:36:41.675 INFO:teuthology.orchestra.run.smithi031.stdout:(4/4): kernel-modules-core-5.14.0-388.el9.x86_6 21 MB/s | 32 MB 00:01 2023-11-29T08:36:41.675 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-11-29T08:36:41.675 INFO:teuthology.orchestra.run.smithi031.stdout:Total 43 MB/s | 96 MB 00:02 2023-11-29T08:36:42.571 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-11-29T08:36:43.002 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-11-29T08:36:43.003 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-11-29T08:36:44.388 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-11-29T08:36:44.388 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-11-29T08:36:47.190 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-11-29T08:36:47.858 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-modules-core-5.14.0-388.el9.x86_64 1/4 2023-11-29T08:36:48.025 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-11-29T08:36:50.909 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 2/4 2023-11-29T08:36:51.092 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-11-29T08:36:56.059 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-11-29T08:36:56.176 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-5.14.0-388.el9.x86_64 4/4 2023-11-29T08:37:01.161 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-11-29T08:37:31.766 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 4/4 2023-11-29T08:37:31.810 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 4/4 2023-11-29T08:37:32.296 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-5.14.0-388.el9.x86_64 4/4 2023-11-29T08:37:32.296 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-5.14.0-388.el9.x86_64 1/4 2023-11-29T08:37:32.296 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-11-29T08:37:32.296 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-11-29T08:37:32.719 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-11-29T08:37:32.719 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:37:32.719 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-11-29T08:37:32.719 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-5.14.0-388.el9.x86_64 2023-11-29T08:37:32.719 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-core-5.14.0-388.el9.x86_64 2023-11-29T08:37:32.719 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-5.14.0-388.el9.x86_64 2023-11-29T08:37:32.719 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-core-5.14.0-388.el9.x86_64 2023-11-29T08:37:32.719 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:37:32.719 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:37:33.239 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2023-11-29T08:37:33.380 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-388.el9.x86_64 2023-11-29T08:37:33.381 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-11-29T08:37:33.381 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-388.el9.x86_64 2023-11-29T08:37:33.381 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-29T08:37:33.381 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-29T08:37:33.381 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-11-29T08:37:33.381 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi031.front.sepia.ceph.com, path=None, version=distro) 2023-11-29T08:37:33.382 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2023-11-29T08:37:33.505 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-388.el9.x86_64 2023-11-29T08:37:33.505 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-11-29T08:37:33.505 DEBUG:teuthology.orchestra.run.smithi031:> sudo rpm -qi grub2-tools 2023-11-29T08:37:33.551 INFO:teuthology.orchestra.run.smithi031.stdout:Name : grub2-tools 2023-11-29T08:37:33.551 INFO:teuthology.orchestra.run.smithi031.stdout:Epoch : 1 2023-11-29T08:37:33.551 INFO:teuthology.orchestra.run.smithi031.stdout:Version : 2.06 2023-11-29T08:37:33.551 INFO:teuthology.orchestra.run.smithi031.stdout:Release : 68.el9 2023-11-29T08:37:33.551 INFO:teuthology.orchestra.run.smithi031.stdout:Architecture: x86_64 2023-11-29T08:37:33.551 INFO:teuthology.orchestra.run.smithi031.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-11-29T08:37:33.551 INFO:teuthology.orchestra.run.smithi031.stdout:Group : Unspecified 2023-11-29T08:37:33.551 INFO:teuthology.orchestra.run.smithi031.stdout:Size : 8264254 2023-11-29T08:37:33.551 INFO:teuthology.orchestra.run.smithi031.stdout:License : GPLv3+ 2023-11-29T08:37:33.552 INFO:teuthology.orchestra.run.smithi031.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-11-29T08:37:33.552 INFO:teuthology.orchestra.run.smithi031.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-11-29T08:37:33.552 INFO:teuthology.orchestra.run.smithi031.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-11-29T08:37:33.552 INFO:teuthology.orchestra.run.smithi031.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-11-29T08:37:33.552 INFO:teuthology.orchestra.run.smithi031.stdout:Packager : builder@centos.org 2023-11-29T08:37:33.552 INFO:teuthology.orchestra.run.smithi031.stdout:Vendor : CentOS 2023-11-29T08:37:33.552 INFO:teuthology.orchestra.run.smithi031.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-29T08:37:33.552 INFO:teuthology.orchestra.run.smithi031.stdout:Summary : Support tools for GRUB. 2023-11-29T08:37:33.552 INFO:teuthology.orchestra.run.smithi031.stdout:Description : 2023-11-29T08:37:33.552 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:37:33.552 INFO:teuthology.orchestra.run.smithi031.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-29T08:37:33.552 INFO:teuthology.orchestra.run.smithi031.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-29T08:37:33.552 INFO:teuthology.orchestra.run.smithi031.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-29T08:37:33.552 INFO:teuthology.orchestra.run.smithi031.stdout:hardware devices. 2023-11-29T08:37:33.552 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:37:33.553 INFO:teuthology.orchestra.run.smithi031.stdout:This subpackage provides tools for support of all platforms. 2023-11-29T08:37:33.553 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-29T08:37:33.553 INFO:teuthology.task.kernel:Updating grub on smithi031 to boot 5.14.0-388.el9.x86_64 2023-11-29T08:37:33.553 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-29T08:37:34.078 INFO:teuthology.orchestra.run.smithi031.stderr:Generating grub configuration file ... 2023-11-29T08:37:35.389 INFO:teuthology.orchestra.run.smithi031.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-11-29T08:37:35.423 INFO:teuthology.orchestra.run.smithi031.stderr:done 2023-11-29T08:37:35.426 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2023-11-29T08:37:35.442 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.Teu6yNPLNm 2023-11-29T08:37:35.442 DEBUG:teuthology.orchestra.run.smithi031:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Teu6yNPLNm 2023-11-29T08:37:35.515 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.Teu6yNPLNm 2023-11-29T08:37:35.661 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.Teu6yNPLNm is 6KB 2023-11-29T08:37:35.711 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.Teu6yNPLNm 2023-11-29T08:37:35.726 DEBUG:teuthology.orchestra.run.smithi031:> sudo /bin/ls /boot/loader/entries || true 2023-11-29T08:37:35.802 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-11-29T08:37:35.802 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-11-29T08:37:35.802 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-11-29T08:37:35.802 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64.conf 2023-11-29T08:37:35.804 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64 2023-11-29T08:37:35.962 DEBUG:teuthology.orchestra.run.smithi031:> sudo shutdown -r now 2023-11-29T08:38:05.994 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-11-29T08:38:05.995 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-11-29T08:38:05.995 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-29T08:39:06.009 DEBUG:teuthology.orchestra.remote:timed out 2023-11-29T08:39:15.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-11-29T08:39:15.019 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-29T08:39:15.398 DEBUG:teuthology.orchestra.run.smithi031:> true 2023-11-29T08:39:15.762 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-11-29T08:39:15.763 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-388.el9.x86_64"... 2023-11-29T08:39:15.763 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2023-11-29T08:39:15.778 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-388.el9.x86_64 2023-11-29T08:39:15.779 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-388.el9.x86_64 vs 5.14.0-388.el9.x86_64 2023-11-29T08:39:15.779 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-29T08:39:15.779 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-29T08:39:16.779 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-11-29T08:39:16.780 DEBUG:teuthology.orchestra.run.smithi031:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-29T08:39:16.853 INFO:teuthology.orchestra.run.smithi031.stdout:ttyS1 2023-11-29T08:39:16.887 DEBUG:teuthology.parallel:result is None 2023-11-29T08:39:16.888 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-29T08:39:16.899 INFO:teuthology.task.internal:Creating test directory... 2023-11-29T08:39:16.899 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-29T08:39:16.917 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-29T08:39:16.924 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-29T08:39:16.932 INFO:teuthology.task.internal:Creating archive directory... 2023-11-29T08:39:16.932 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-29T08:39:17.000 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-29T08:39:17.007 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-29T08:39:17.007 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-29T08:39:17.082 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-29T08:39:17.099 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-29T08:39:17.101 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-29T08:39:17.108 INFO:teuthology.task.internal:Configuring sudo... 2023-11-29T08:39:17.108 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-29T08:39:17.142 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-29T08:39:17.167 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-29T08:39:17.168 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-29T08:39:17.199 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-29T08:39:17.290 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-29T08:39:17.337 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-29T08:39:17.428 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-29T08:39:17.469 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-11-29T08:39:17.469 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-29T08:39:17.542 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2023-11-29T08:39:17.633 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-29T08:39:17.856 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-29T08:39:17.864 INFO:teuthology.task.internal:Starting timer... 2023-11-29T08:39:17.864 INFO:teuthology.run_tasks:Running task pcp... 2023-11-29T08:39:17.884 INFO:teuthology.run_tasks:Running task selinux... 2023-11-29T08:39:17.893 DEBUG:teuthology.orchestra.run.smithi031:> sudo service auditd rotate 2023-11-29T08:39:17.950 INFO:teuthology.orchestra.run.smithi031.stdout:Rotating logs: 2023-11-29T08:39:17.952 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-29T08:39:17.952 DEBUG:teuthology.orchestra.run.smithi031:> /usr/sbin/getenforce 2023-11-29T08:39:17.982 INFO:teuthology.orchestra.run.smithi031.stdout:Permissive 2023-11-29T08:39:17.982 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi031.front.sepia.ceph.com': 'permissive'} 2023-11-29T08:39:17.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-29T08:39:18.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-29T08:39:18.055 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-29T08:39:18.055 DEBUG:teuthology.orchestra.run.smithi031:> sudo /usr/sbin/setenforce permissive 2023-11-29T08:39:18.133 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-29T08:39:18.143 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-11-29T08:39:18.149 INFO:teuthology.repo_utils:Fetching main from origin 2023-11-29T08:39:18.307 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-29T08:39:18.324 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-29T08:39:18.325 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi031.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-29T08:45:33.594 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi031.front.sepia.ceph.com')] 2023-11-29T08:45:33.595 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-11-29T08:45:33.596 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-29T08:45:33.737 DEBUG:teuthology.orchestra.run.smithi031:> true 2023-11-29T08:45:33.806 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-11-29T08:45:33.807 INFO:teuthology.run_tasks:Running task clock... 2023-11-29T08:45:33.822 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-29T08:45:33.822 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-29T08:45:33.823 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-29T08:45:33.882 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-29T08:45:33.903 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-29T08:45:33.939 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: ntpd: command not found 2023-11-29T08:45:33.956 INFO:teuthology.orchestra.run.smithi031.stdout:506 Cannot talk to daemon 2023-11-29T08:45:33.975 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-29T08:45:33.995 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-29T08:45:34.042 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2023-11-29T08:45:34.046 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-29T08:45:34.046 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2023-11-29T08:45:34.046 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-29T08:45:34.046 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-29T08:45:34.046 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-29T08:45:34.046 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-29T08:45:34.047 INFO:teuthology.run_tasks:Running task install... 2023-11-29T08:45:34.058 DEBUG:teuthology.task.install:project ceph 2023-11-29T08:45:34.058 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '412bfade5c44735b71d7c14eb55649269cd32a35'}} 2023-11-29T08:45:34.058 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '412bfade5c44735b71d7c14eb55649269cd32a35'} 2023-11-29T08:45:34.058 INFO:teuthology.task.install:Using flavor: crimson 2023-11-29T08:45:34.064 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-29T08:45:34.064 INFO:teuthology.task.install:extra packages: [] 2023-11-29T08:45:34.065 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '412bfade5c44735b71d7c14eb55649269cd32a35', 'tag': None, 'wait_for_package': False} 2023-11-29T08:45:34.065 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=412bfade5c44735b71d7c14eb55649269cd32a35 2023-11-29T08:45:34.293 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-snaptrimevent-lifetime-v2/412bfade5c44735b71d7c14eb55649269cd32a35/centos/9/flavors/crimson/ 2023-11-29T08:45:34.293 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7483.g412bfade 2023-11-29T08:45:34.418 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-snaptrimevent-lifetime-v2/412bfade5c44735b71d7c14eb55649269cd32a35/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-snaptrimevent-lifetime-v2/412bfade5c44735b71d7c14eb55649269cd32a35/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-snaptrimevent-lifetime-v2/412bfade5c44735b71d7c14eb55649269cd32a35/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-29T08:45:34.418 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-11-29T08:45:34.418 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-29T08:45:34.450 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-29T08:45:34.451 DEBUG:teuthology.orchestra.run.smithi031:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/412bfade5c44735b71d7c14eb55649269cd32a35/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-29T08:45:34.525 DEBUG:teuthology.orchestra.run.smithi031:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-29T08:45:34.613 DEBUG:teuthology.orchestra.run.smithi031:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-29T08:45:34.643 INFO:teuthology.orchestra.run.smithi031.stdout:check_obsoletes = 1 2023-11-29T08:45:34.644 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum clean all 2023-11-29T08:45:34.901 INFO:teuthology.orchestra.run.smithi031.stdout:73 files removed 2023-11-29T08:45:34.928 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-radosgw 2023-11-29T08:45:36.072 INFO:teuthology.orchestra.run.smithi031.stdout:ceph packages for x86_64 101 kB/s | 81 kB 00:00 2023-11-29T08:45:36.481 INFO:teuthology.orchestra.run.smithi031.stdout:ceph noarch packages 27 kB/s | 9.5 kB 00:00 2023-11-29T08:45:36.823 INFO:teuthology.orchestra.run.smithi031.stdout:ceph source packages 6.5 kB/s | 1.9 kB 00:00 2023-11-29T08:45:37.425 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - AppStream 11 MB/s | 6.4 MB 00:00 2023-11-29T08:45:39.050 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2023-11-29T08:45:39.858 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 7.9 MB 00:00 2023-11-29T08:45:41.784 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 24 MB/s | 18 MB 00:00 2023-11-29T08:45:45.751 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 15 MB/s | 5.8 MB 00:00 2023-11-29T08:45:47.302 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 15 kB 00:00 2023-11-29T08:45:49.323 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2023-11-29T08:45:54.187 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2023-11-29T08:45:57.007 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:45:57.020 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:45:57.021 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-11-29T08:45:57.021 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:45:57.021 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-11-29T08:45:57.021 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw x86_64 2:18.0.0-7483.g412bfade.el9 ceph 8.4 M 2023-11-29T08:45:57.021 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrading: 2023-11-29T08:45:57.021 INFO:teuthology.orchestra.run.smithi031.stdout: librados2 x86_64 2:18.0.0-7483.g412bfade.el9 ceph 3.1 M 2023-11-29T08:45:57.021 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1 x86_64 2:18.0.0-7483.g412bfade.el9 ceph 3.0 M 2023-11-29T08:45:57.021 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-11-29T08:45:57.021 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options 2023-11-29T08:45:57.021 INFO:teuthology.orchestra.run.smithi031.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-11-29T08:45:57.021 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base x86_64 2:18.0.0-7483.g412bfade.el9 ceph 6.6 M 2023-11-29T08:45:57.022 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common x86_64 2:18.0.0-7483.g412bfade.el9 ceph 17 M 2023-11-29T08:45:57.022 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux x86_64 2:18.0.0-7483.g412bfade.el9 ceph 24 k 2023-11-29T08:45:57.022 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-11-29T08:45:57.022 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-11-29T08:45:57.022 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-11-29T08:45:57.022 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2 x86_64 2:18.0.0-7483.g412bfade.el9 ceph 728 k 2023-11-29T08:45:57.022 INFO:teuthology.orchestra.run.smithi031.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-11-29T08:45:57.022 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-11-29T08:45:57.022 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-11-29T08:45:57.022 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1 x86_64 2:18.0.0-7483.g412bfade.el9 ceph 470 k 2023-11-29T08:45:57.022 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-11-29T08:45:57.023 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2 x86_64 2:18.0.0-7483.g412bfade.el9 ceph 4.5 M 2023-11-29T08:45:57.023 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-11-29T08:45:57.023 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-11-29T08:45:57.023 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-11-29T08:45:57.023 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse 2023-11-29T08:45:57.023 INFO:teuthology.orchestra.run.smithi031.stdout: x86_64 2:18.0.0-7483.g412bfade.el9 ceph 45 k 2023-11-29T08:45:57.023 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common x86_64 2:18.0.0-7483.g412bfade.el9 ceph 129 k 2023-11-29T08:45:57.023 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs x86_64 2:18.0.0-7483.g412bfade.el9 ceph 161 k 2023-11-29T08:45:57.023 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-11-29T08:45:57.023 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados x86_64 2:18.0.0-7483.g412bfade.el9 ceph 321 k 2023-11-29T08:45:57.023 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd x86_64 2:18.0.0-7483.g412bfade.el9 ceph 298 k 2023-11-29T08:45:57.023 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw x86_64 2:18.0.0-7483.g412bfade.el9 ceph 99 k 2023-11-29T08:45:57.024 INFO:teuthology.orchestra.run.smithi031.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-11-29T08:45:57.024 INFO:teuthology.orchestra.run.smithi031.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-11-29T08:45:57.024 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:45:57.024 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-11-29T08:45:57.024 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:45:57.024 INFO:teuthology.orchestra.run.smithi031.stdout:Install 27 Packages 2023-11-29T08:45:57.024 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrade 2 Packages 2023-11-29T08:45:57.024 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:45:57.025 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 54 M 2023-11-29T08:45:57.025 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-11-29T08:45:58.013 INFO:teuthology.orchestra.run.smithi031.stdout:(1/29): ceph-base-18.0.0-7483.g412bfade.el9.x86 8.2 MB/s | 6.6 MB 00:00 2023-11-29T08:45:58.080 INFO:teuthology.orchestra.run.smithi031.stdout:(2/29): ceph-selinux-18.0.0-7483.g412bfade.el9. 361 kB/s | 24 kB 00:00 2023-11-29T08:45:58.197 INFO:teuthology.orchestra.run.smithi031.stdout:(3/29): libcephfs2-18.0.0-7483.g412bfade.el9.x8 6.1 MB/s | 728 kB 00:00 2023-11-29T08:45:58.297 INFO:teuthology.orchestra.run.smithi031.stdout:(4/29): libradosstriper1-18.0.0-7483.g412bfade. 4.6 MB/s | 470 kB 00:00 2023-11-29T08:45:58.705 INFO:teuthology.orchestra.run.smithi031.stdout:(5/29): librgw2-18.0.0-7483.g412bfade.el9.x86_6 11 MB/s | 4.5 MB 00:00 2023-11-29T08:45:58.814 INFO:teuthology.orchestra.run.smithi031.stdout:(6/29): ceph-radosgw-18.0.0-7483.g412bfade.el9. 5.2 MB/s | 8.4 MB 00:01 2023-11-29T08:45:58.839 INFO:teuthology.orchestra.run.smithi031.stdout:(7/29): python3-ceph-argparse-18.0.0-7483.g412b 338 kB/s | 45 kB 00:00 2023-11-29T08:45:58.923 INFO:teuthology.orchestra.run.smithi031.stdout:(8/29): python3-ceph-common-18.0.0-7483.g412bfa 1.2 MB/s | 129 kB 00:00 2023-11-29T08:45:58.956 INFO:teuthology.orchestra.run.smithi031.stdout:(9/29): python3-cephfs-18.0.0-7483.g412bfade.el 1.3 MB/s | 161 kB 00:00 2023-11-29T08:45:59.006 INFO:teuthology.orchestra.run.smithi031.stdout:(10/29): python3-rados-18.0.0-7483.g412bfade.el 3.8 MB/s | 321 kB 00:00 2023-11-29T08:45:59.049 INFO:teuthology.orchestra.run.smithi031.stdout:(11/29): python3-rbd-18.0.0-7483.g412bfade.el9. 3.2 MB/s | 298 kB 00:00 2023-11-29T08:45:59.107 INFO:teuthology.orchestra.run.smithi031.stdout:(12/29): python3-rgw-18.0.0-7483.g412bfade.el9. 986 kB/s | 99 kB 00:00 2023-11-29T08:45:59.174 INFO:teuthology.orchestra.run.smithi031.stdout:(13/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.4 MB/s | 159 kB 00:00 2023-11-29T08:45:59.200 INFO:teuthology.orchestra.run.smithi031.stdout:(14/29): boost-program-options-1.75.0-8.el9.x86 703 kB/s | 106 kB 00:00 2023-11-29T08:45:59.225 INFO:teuthology.orchestra.run.smithi031.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 880 kB/s | 44 kB 00:00 2023-11-29T08:45:59.275 INFO:teuthology.orchestra.run.smithi031.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2023-11-29T08:45:59.301 INFO:teuthology.orchestra.run.smithi031.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2023-11-29T08:45:59.326 INFO:teuthology.orchestra.run.smithi031.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 827 kB/s | 41 kB 00:00 2023-11-29T08:45:59.843 INFO:teuthology.orchestra.run.smithi031.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 570 kB/s | 309 kB 00:00 2023-11-29T08:45:59.951 INFO:teuthology.orchestra.run.smithi031.stdout:(20/29): libarrow-doc-9.0.0-9.el9.noarch.rpm 225 kB/s | 24 kB 00:00 2023-11-29T08:46:00.060 INFO:teuthology.orchestra.run.smithi031.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2023-11-29T08:46:00.144 INFO:teuthology.orchestra.run.smithi031.stdout:(22/29): libarrow-9.0.0-9.el9.x86_64.rpm 5.4 MB/s | 4.4 MB 00:00 2023-11-29T08:46:00.161 INFO:teuthology.orchestra.run.smithi031.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 672 kB/s | 67 kB 00:00 2023-11-29T08:46:00.353 INFO:teuthology.orchestra.run.smithi031.stdout:(24/29): ceph-common-18.0.0-7483.g412bfade.el9. 5.5 MB/s | 17 MB 00:03 2023-11-29T08:46:00.378 INFO:teuthology.orchestra.run.smithi031.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 882 kB/s | 192 kB 00:00 2023-11-29T08:46:00.411 INFO:teuthology.orchestra.run.smithi031.stdout:(26/29): parquet-libs-9.0.0-9.el9.x86_64.rpm 3.1 MB/s | 837 kB 00:00 2023-11-29T08:46:00.728 INFO:teuthology.orchestra.run.smithi031.stdout:(27/29): librados2-18.0.0-7483.g412bfade.el9.x8 8.9 MB/s | 3.1 MB 00:00 2023-11-29T08:46:00.828 INFO:teuthology.orchestra.run.smithi031.stdout:(28/29): librbd1-18.0.0-7483.g412bfade.el9.x86_ 7.2 MB/s | 3.0 MB 00:00 2023-11-29T08:46:01.171 INFO:teuthology.orchestra.run.smithi031.stdout:(29/29): thrift-0.15.0-2.el9.x86_64.rpm 1.9 MB/s | 1.6 MB 00:00 2023-11-29T08:46:01.174 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-11-29T08:46:01.174 INFO:teuthology.orchestra.run.smithi031.stdout:Total 13 MB/s | 54 MB 00:04 2023-11-29T08:46:01.174 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-11-29T08:46:01.214 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-11-29T08:46:01.215 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-11-29T08:46:01.691 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-11-29T08:46:01.691 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-11-29T08:46:03.222 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-11-29T08:46:03.632 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-11-29T08:46:03.681 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librados2-2:18.0.0-7483.g412bfade.el9.x86_64 2/31 2023-11-29T08:46:03.774 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:18.0.0-7483.g412bfade.el9.x86_64 2/31 2023-11-29T08:46:03.873 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rados-2:18.0.0-7483.g412bfade.el9.x86_64 3/31 2023-11-29T08:46:03.917 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs2-2:18.0.0-7483.g412bfade.el9.x86_64 4/31 2023-11-29T08:46:04.001 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs2-2:18.0.0-7483.g412bfade.el9.x86_64 4/31 2023-11-29T08:46:04.037 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/31 2023-11-29T08:46:04.063 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/31 2023-11-29T08:46:04.093 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/31 2023-11-29T08:46:04.137 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-argparse-2:18.0.0-7483.g412bfade.el9. 8/31 2023-11-29T08:46:04.205 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cephfs-2:18.0.0-7483.g412bfade.el9.x86_64 9/31 2023-11-29T08:46:04.229 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libradosstriper1-2:18.0.0-7483.g412bfade.el9.x86_6 10/31 2023-11-29T08:46:04.318 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7483.g412bfade.el9.x86_6 10/31 2023-11-29T08:46:04.353 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/31 2023-11-29T08:46:04.465 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-11-29T08:46:04.496 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/31 2023-11-29T08:46:04.537 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-11-29T08:46:04.577 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-11-29T08:46:04.606 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/31 2023-11-29T08:46:04.639 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-11-29T08:46:04.968 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-11-29T08:46:04.993 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librbd1-2:18.0.0-7483.g412bfade.el9.x86_64 19/31 2023-11-29T08:46:05.084 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:18.0.0-7483.g412bfade.el9.x86_64 19/31 2023-11-29T08:46:05.118 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rbd-2:18.0.0-7483.g412bfade.el9.x86_64 20/31 2023-11-29T08:46:05.159 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-11-29T08:46:05.613 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 22/31 2023-11-29T08:46:05.638 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librgw2-2:18.0.0-7483.g412bfade.el9.x86_64 23/31 2023-11-29T08:46:05.717 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librgw2-2:18.0.0-7483.g412bfade.el9.x86_64 23/31 2023-11-29T08:46:05.788 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rgw-2:18.0.0-7483.g412bfade.el9.x86_64 24/31 2023-11-29T08:46:05.823 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-common-2:18.0.0-7483.g412bfade.el9.x8 25/31 2023-11-29T08:46:07.966 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:18.0.0-7483.g412bfade.el9.x86_64 26/31 2023-11-29T08:46:07.999 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-common-2:18.0.0-7483.g412bfade.el9.x86_64 26/31 2023-11-29T08:46:08.794 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:18.0.0-7483.g412bfade.el9.x86_64 26/31 2023-11-29T08:46:08.919 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-base-2:18.0.0-7483.g412bfade.el9.x86_64 27/31 2023-11-29T08:46:09.011 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-base-2:18.0.0-7483.g412bfade.el9.x86_64 27/31 2023-11-29T08:46:09.011 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-11-29T08:46:09.011 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-11-29T08:46:09.011 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:46:09.045 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-selinux-2:18.0.0-7483.g412bfade.el9.x86_64 28/31 2023-11-29T08:46:19.254 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7483.g412bfade.el9.x86_64 28/31 2023-11-29T08:46:19.255 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /sys 2023-11-29T08:46:19.255 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /proc 2023-11-29T08:46:19.255 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /mnt 2023-11-29T08:46:19.255 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /var/tmp 2023-11-29T08:46:19.255 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /home 2023-11-29T08:46:19.255 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /root 2023-11-29T08:46:19.255 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /tmp 2023-11-29T08:46:19.255 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:46:20.117 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-radosgw-2:18.0.0-7483.g412bfade.el9.x86_64 29/31 2023-11-29T08:46:20.143 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7483.g412bfade.el9.x86_64 29/31 2023-11-29T08:46:20.143 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-29T08:46:20.143 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-11-29T08:46:20.143 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-29T08:46:20.143 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-29T08:46:20.143 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:46:20.145 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-11-29T08:46:20.227 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-11-29T08:46:20.227 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-11-29T08:46:21.418 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-11-29T08:46:21.418 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-base-2:18.0.0-7483.g412bfade.el9.x86_64 1/31 2023-11-29T08:46:21.418 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-common-2:18.0.0-7483.g412bfade.el9.x86_64 2/31 2023-11-29T08:46:21.418 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-radosgw-2:18.0.0-7483.g412bfade.el9.x86_64 3/31 2023-11-29T08:46:21.419 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-selinux-2:18.0.0-7483.g412bfade.el9.x86_64 4/31 2023-11-29T08:46:21.419 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs2-2:18.0.0-7483.g412bfade.el9.x86_64 5/31 2023-11-29T08:46:21.419 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libradosstriper1-2:18.0.0-7483.g412bfade.el9.x86_6 6/31 2023-11-29T08:46:21.419 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librgw2-2:18.0.0-7483.g412bfade.el9.x86_64 7/31 2023-11-29T08:46:21.419 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7483.g412bfade.el9. 8/31 2023-11-29T08:46:21.419 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-common-2:18.0.0-7483.g412bfade.el9.x8 9/31 2023-11-29T08:46:21.419 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cephfs-2:18.0.0-7483.g412bfade.el9.x86_64 10/31 2023-11-29T08:46:21.419 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rados-2:18.0.0-7483.g412bfade.el9.x86_64 11/31 2023-11-29T08:46:21.419 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rbd-2:18.0.0-7483.g412bfade.el9.x86_64 12/31 2023-11-29T08:46:21.419 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rgw-2:18.0.0-7483.g412bfade.el9.x86_64 13/31 2023-11-29T08:46:21.419 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-11-29T08:46:21.419 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-11-29T08:46:21.419 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-11-29T08:46:21.419 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-11-29T08:46:21.419 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-11-29T08:46:21.420 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-11-29T08:46:21.420 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-11-29T08:46:21.420 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 21/31 2023-11-29T08:46:21.420 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 22/31 2023-11-29T08:46:21.420 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-11-29T08:46:21.420 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-11-29T08:46:21.420 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 25/31 2023-11-29T08:46:21.420 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-11-29T08:46:21.420 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-11-29T08:46:21.420 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:18.0.0-7483.g412bfade.el9.x86_64 28/31 2023-11-29T08:46:21.420 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-11-29T08:46:21.420 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:18.0.0-7483.g412bfade.el9.x86_64 30/31 2023-11-29T08:46:22.660 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-11-29T08:46:22.660 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:46:22.660 INFO:teuthology.orchestra.run.smithi031.stdout:Upgraded: 2023-11-29T08:46:22.660 INFO:teuthology.orchestra.run.smithi031.stdout: librados2-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:46:22.660 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:46:22.660 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-11-29T08:46:22.660 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-11-29T08:46:22.661 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:46:22.661 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:46:22.661 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:46:22.661 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:46:22.661 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-11-29T08:46:22.661 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-9.0.0-9.el9.x86_64 2023-11-29T08:46:22.661 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-11-29T08:46:22.661 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:46:22.661 INFO:teuthology.orchestra.run.smithi031.stdout: liboath-2.6.7-2.el9.x86_64 2023-11-29T08:46:22.661 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-11-29T08:46:22.661 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-11-29T08:46:22.661 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:46:22.661 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-11-29T08:46:22.662 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:46:22.662 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind-1.6.2-1.el9.x86_64 2023-11-29T08:46:22.662 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-11-29T08:46:22.662 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-11-29T08:46:22.662 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:46:22.662 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:46:22.662 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:46:22.662 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-11-29T08:46:22.662 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:46:22.662 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:46:22.662 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:46:22.662 INFO:teuthology.orchestra.run.smithi031.stdout: re2-1:20211101-3.el9.x86_64 2023-11-29T08:46:22.662 INFO:teuthology.orchestra.run.smithi031.stdout: thrift-0.15.0-2.el9.x86_64 2023-11-29T08:46:22.662 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:46:22.662 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:46:22.991 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-test 2023-11-29T08:46:23.591 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:29 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:46:23.754 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:46:23.755 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:46:23.755 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-11-29T08:46:23.755 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:46:23.755 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-11-29T08:46:23.755 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test x86_64 2:18.0.0-7483.g412bfade.el9 ceph 118 M 2023-11-29T08:46:23.755 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-11-29T08:46:23.756 INFO:teuthology.orchestra.run.smithi031.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2023-11-29T08:46:23.756 INFO:teuthology.orchestra.run.smithi031.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-11-29T08:46:23.756 INFO:teuthology.orchestra.run.smithi031.stdout: libasan x86_64 11.4.1-2.3.el9 CentOS-AppStream 418 k 2023-11-29T08:46:23.756 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite x86_64 2:18.0.0-7483.g412bfade.el9 ceph 158 k 2023-11-29T08:46:23.756 INFO:teuthology.orchestra.run.smithi031.stdout: libubsan x86_64 11.4.1-2.3.el9 CentOS-AppStream 186 k 2023-11-29T08:46:23.756 INFO:teuthology.orchestra.run.smithi031.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-11-29T08:46:23.756 INFO:teuthology.orchestra.run.smithi031.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-11-29T08:46:23.756 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-11-29T08:46:23.756 INFO:teuthology.orchestra.run.smithi031.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2023-11-29T08:46:23.756 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:46:23.756 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-11-29T08:46:23.756 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:46:23.756 INFO:teuthology.orchestra.run.smithi031.stdout:Install 10 Packages 2023-11-29T08:46:23.756 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:46:23.757 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 121 M 2023-11-29T08:46:23.757 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 1.1 G 2023-11-29T08:46:23.757 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-11-29T08:46:24.015 INFO:teuthology.orchestra.run.smithi031.stdout:(1/10): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 186 kB 00:00 2023-11-29T08:46:24.132 INFO:teuthology.orchestra.run.smithi031.stdout:(2/10): libasan-11.4.1-2.3.el9.x86_64.rpm 3.5 MB/s | 418 kB 00:00 2023-11-29T08:46:24.190 INFO:teuthology.orchestra.run.smithi031.stdout:(3/10): libubsan-11.4.1-2.3.el9.x86_64.rpm 3.1 MB/s | 186 kB 00:00 2023-11-29T08:46:24.249 INFO:teuthology.orchestra.run.smithi031.stdout:(4/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2023-11-29T08:46:24.316 INFO:teuthology.orchestra.run.smithi031.stdout:(5/10): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 300 kB 00:00 2023-11-29T08:46:24.359 INFO:teuthology.orchestra.run.smithi031.stdout:(6/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2023-11-29T08:46:24.517 INFO:teuthology.orchestra.run.smithi031.stdout:(7/10): libcephsqlite-18.0.0-7483.g412bfade.el9 233 kB/s | 158 kB 00:00 2023-11-29T08:46:24.742 INFO:teuthology.orchestra.run.smithi031.stdout:(8/10): cryptopp-8.6.0-1.el9.x86_64.rpm 3.3 MB/s | 1.3 MB 00:00 2023-11-29T08:46:24.775 INFO:teuthology.orchestra.run.smithi031.stdout:(9/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 472 kB/s | 122 kB 00:00 2023-11-29T08:46:42.059 INFO:teuthology.orchestra.run.smithi031.stdout:(10/10): ceph-test-18.0.0-7483.g412bfade.el9.x8 6.5 MB/s | 118 MB 00:18 2023-11-29T08:46:42.062 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-11-29T08:46:42.063 INFO:teuthology.orchestra.run.smithi031.stdout:Total 6.6 MB/s | 121 MB 00:18 2023-11-29T08:46:42.063 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-11-29T08:46:42.088 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-11-29T08:46:42.088 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-11-29T08:46:42.946 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-11-29T08:46:42.946 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-11-29T08:46:44.094 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-11-29T08:46:44.192 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2023-11-29T08:46:44.232 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2023-11-29T08:46:44.284 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2023-11-29T08:46:44.317 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2023-11-29T08:46:44.350 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-11-29T08:46:44.381 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : jq-1.6-16.el9.x86_64 6/10 2023-11-29T08:46:44.415 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libubsan-11.4.1-2.3.el9.x86_64 7/10 2023-11-29T08:46:44.452 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libasan-11.4.1-2.3.el9.x86_64 8/10 2023-11-29T08:46:44.470 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephsqlite-2:18.0.0-7483.g412bfade.el9.x86_64 9/10 2023-11-29T08:47:01.603 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7483.g412bfade.el9.x86_64 9/10 2023-11-29T08:47:01.682 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-test-2:18.0.0-7483.g412bfade.el9.x86_64 10/10 2023-11-29T08:47:09.652 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-test-2:18.0.0-7483.g412bfade.el9.x86_64 10/10 2023-11-29T08:47:09.652 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-test-2:18.0.0-7483.g412bfade.el9.x86_64 1/10 2023-11-29T08:47:09.652 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephsqlite-2:18.0.0-7483.g412bfade.el9.x86_64 2/10 2023-11-29T08:47:09.653 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : jq-1.6-16.el9.x86_64 3/10 2023-11-29T08:47:09.653 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libasan-11.4.1-2.3.el9.x86_64 4/10 2023-11-29T08:47:09.653 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libubsan-11.4.1-2.3.el9.x86_64 5/10 2023-11-29T08:47:09.653 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/10 2023-11-29T08:47:09.653 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 7/10 2023-11-29T08:47:09.653 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/10 2023-11-29T08:47:09.653 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2023-11-29T08:47:10.131 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2023-11-29T08:47:10.132 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:47:10.132 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-11-29T08:47:10.132 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:47:10.132 INFO:teuthology.orchestra.run.smithi031.stdout: cryptopp-8.6.0-1.el9.x86_64 2023-11-29T08:47:10.132 INFO:teuthology.orchestra.run.smithi031.stdout: jq-1.6-16.el9.x86_64 2023-11-29T08:47:10.132 INFO:teuthology.orchestra.run.smithi031.stdout: libasan-11.4.1-2.3.el9.x86_64 2023-11-29T08:47:10.132 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:47:10.132 INFO:teuthology.orchestra.run.smithi031.stdout: libubsan-11.4.1-2.3.el9.x86_64 2023-11-29T08:47:10.132 INFO:teuthology.orchestra.run.smithi031.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-11-29T08:47:10.132 INFO:teuthology.orchestra.run.smithi031.stdout: socat-1.7.4.1-5.el9.x86_64 2023-11-29T08:47:10.132 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-11-29T08:47:10.132 INFO:teuthology.orchestra.run.smithi031.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2023-11-29T08:47:10.132 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:47:10.132 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:47:10.437 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph 2023-11-29T08:47:11.038 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:17 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:47:11.201 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:47:11.203 INFO:teuthology.orchestra.run.smithi031.stdout:========================================================================================= 2023-11-29T08:47:11.203 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-11-29T08:47:11.203 INFO:teuthology.orchestra.run.smithi031.stdout:========================================================================================= 2023-11-29T08:47:11.203 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-11-29T08:47:11.203 INFO:teuthology.orchestra.run.smithi031.stdout: ceph x86_64 2:18.0.0-7483.g412bfade.el9 ceph 6.5 k 2023-11-29T08:47:11.203 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-11-29T08:47:11.204 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds x86_64 2:18.0.0-7483.g412bfade.el9 ceph 2.0 M 2023-11-29T08:47:11.204 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr x86_64 2:18.0.0-7483.g412bfade.el9 ceph 1.5 M 2023-11-29T08:47:11.204 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7483.g412bfade.el9 ceph-noarch 243 k 2023-11-29T08:47:11.204 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon x86_64 2:18.0.0-7483.g412bfade.el9 ceph 8.9 M 2023-11-29T08:47:11.204 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd x86_64 2:18.0.0-7483.g412bfade.el9 ceph 117 M 2023-11-29T08:47:11.204 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-11-29T08:47:11.205 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-11-29T08:47:11.205 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-11-29T08:47:11.205 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-11-29T08:47:11.205 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-11-29T08:47:11.205 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-11-29T08:47:11.205 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-11-29T08:47:11.206 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-11-29T08:47:11.206 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-11-29T08:47:11.206 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-11-29T08:47:11.206 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-11-29T08:47:11.206 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-11-29T08:47:11.206 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-11-29T08:47:11.207 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-11-29T08:47:11.207 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-11-29T08:47:11.207 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-11-29T08:47:11.207 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-11-29T08:47:11.207 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-11-29T08:47:11.207 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-11-29T08:47:11.208 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-11-29T08:47:11.208 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-11-29T08:47:11.208 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-11-29T08:47:11.208 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-11-29T08:47:11.208 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-11-29T08:47:11.208 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-11-29T08:47:11.209 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-11-29T08:47:11.209 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-11-29T08:47:11.210 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:47:11.210 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-11-29T08:47:11.210 INFO:teuthology.orchestra.run.smithi031.stdout:========================================================================================= 2023-11-29T08:47:11.211 INFO:teuthology.orchestra.run.smithi031.stdout:Install 33 Packages 2023-11-29T08:47:11.211 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:47:11.212 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 134 M 2023-11-29T08:47:11.212 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 1.4 G 2023-11-29T08:47:11.212 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-11-29T08:47:11.717 INFO:teuthology.orchestra.run.smithi031.stdout:(1/33): ceph-18.0.0-7483.g412bfade.el9.x86_64.r 15 kB/s | 6.5 kB 00:00 2023-11-29T08:47:13.033 INFO:teuthology.orchestra.run.smithi031.stdout:(2/33): ceph-mgr-18.0.0-7483.g412bfade.el9.x86_ 853 kB/s | 1.5 MB 00:01 2023-11-29T08:47:13.692 INFO:teuthology.orchestra.run.smithi031.stdout:(3/33): ceph-mds-18.0.0-7483.g412bfade.el9.x86_ 852 kB/s | 2.0 MB 00:02 2023-11-29T08:47:14.025 INFO:teuthology.orchestra.run.smithi031.stdout:(4/33): ceph-mgr-modules-core-18.0.0-7483.g412b 728 kB/s | 243 kB 00:00 2023-11-29T08:47:14.218 INFO:teuthology.orchestra.run.smithi031.stdout:(5/33): ceph-mon-18.0.0-7483.g412bfade.el9.x86_ 3.5 MB/s | 8.9 MB 00:02 2023-11-29T08:47:14.242 INFO:teuthology.orchestra.run.smithi031.stdout:(6/33): python3-mako-1.1.4-6.el9.noarch.rpm 665 kB/s | 144 kB 00:00 2023-11-29T08:47:14.268 INFO:teuthology.orchestra.run.smithi031.stdout:(7/33): python3-markupsafe-1.1.1-12.el9.x86_64. 632 kB/s | 32 kB 00:00 2023-11-29T08:47:14.293 INFO:teuthology.orchestra.run.smithi031.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 947 kB/s | 47 kB 00:00 2023-11-29T08:47:14.318 INFO:teuthology.orchestra.run.smithi031.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 774 kB/s | 39 kB 00:00 2023-11-29T08:47:14.352 INFO:teuthology.orchestra.run.smithi031.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2023-11-29T08:47:14.435 INFO:teuthology.orchestra.run.smithi031.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 9.8 MB/s | 1.1 MB 00:00 2023-11-29T08:47:14.461 INFO:teuthology.orchestra.run.smithi031.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 953 kB/s | 103 kB 00:00 2023-11-29T08:47:14.486 INFO:teuthology.orchestra.run.smithi031.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2023-11-29T08:47:14.511 INFO:teuthology.orchestra.run.smithi031.stdout:(14/33): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-11-29T08:47:14.536 INFO:teuthology.orchestra.run.smithi031.stdout:(15/33): python3-urllib3-1.26.5-4.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2023-11-29T08:47:14.628 INFO:teuthology.orchestra.run.smithi031.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 373 kB/s | 43 kB 00:00 2023-11-29T08:47:14.653 INFO:teuthology.orchestra.run.smithi031.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2023-11-29T08:47:14.704 INFO:teuthology.orchestra.run.smithi031.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2023-11-29T08:47:14.746 INFO:teuthology.orchestra.run.smithi031.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2023-11-29T08:47:14.780 INFO:teuthology.orchestra.run.smithi031.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 234 kB/s | 18 kB 00:00 2023-11-29T08:47:14.805 INFO:teuthology.orchestra.run.smithi031.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 395 kB/s | 23 kB 00:00 2023-11-29T08:47:14.830 INFO:teuthology.orchestra.run.smithi031.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 389 kB/s | 19 kB 00:00 2023-11-29T08:47:14.847 INFO:teuthology.orchestra.run.smithi031.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 471 kB/s | 20 kB 00:00 2023-11-29T08:47:14.870 INFO:teuthology.orchestra.run.smithi031.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2023-11-29T08:47:14.896 INFO:teuthology.orchestra.run.smithi031.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.6 MB/s | 79 kB 00:00 2023-11-29T08:47:14.929 INFO:teuthology.orchestra.run.smithi031.stdout:(26/33): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2023-11-29T08:47:14.954 INFO:teuthology.orchestra.run.smithi031.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2023-11-29T08:47:14.980 INFO:teuthology.orchestra.run.smithi031.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2023-11-29T08:47:15.005 INFO:teuthology.orchestra.run.smithi031.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2023-11-29T08:47:15.030 INFO:teuthology.orchestra.run.smithi031.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2023-11-29T08:47:15.072 INFO:teuthology.orchestra.run.smithi031.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 481 kB/s | 20 kB 00:00 2023-11-29T08:47:15.123 INFO:teuthology.orchestra.run.smithi031.stdout:(32/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2023-11-29T08:48:14.732 INFO:teuthology.orchestra.run.smithi031.stdout:(33/33): ceph-osd-18.0.0-7483.g412bfade.el9.x86 1.9 MB/s | 117 MB 01:01 2023-11-29T08:48:14.734 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-11-29T08:48:14.734 INFO:teuthology.orchestra.run.smithi031.stdout:Total 2.1 MB/s | 134 MB 01:03 2023-11-29T08:48:14.735 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-11-29T08:48:14.772 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-11-29T08:48:14.772 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-11-29T08:48:15.808 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-11-29T08:48:15.808 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-11-29T08:48:17.792 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-11-29T08:48:17.875 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-11-29T08:48:17.907 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-11-29T08:48:17.941 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-11-29T08:48:18.035 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-11-29T08:48:18.104 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-11-29T08:48:18.152 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-11-29T08:48:18.234 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-11-29T08:48:18.294 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 8/33 2023-11-29T08:48:18.332 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 9/33 2023-11-29T08:48:18.381 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2023-11-29T08:48:18.445 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2023-11-29T08:48:18.646 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2023-11-29T08:48:18.703 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2023-11-29T08:48:18.742 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2023-11-29T08:48:18.776 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2023-11-29T08:48:18.806 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2023-11-29T08:48:18.884 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2023-11-29T08:48:18.916 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-11-29T08:48:18.949 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2023-11-29T08:48:18.990 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2023-11-29T08:48:19.025 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2023-11-29T08:48:19.059 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-11-29T08:48:19.151 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-11-29T08:48:19.190 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-11-29T08:48:19.265 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-11-29T08:48:19.421 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-11-29T08:48:19.543 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-11-29T08:48:19.717 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7483.g412bfade.el9. 28/33 2023-11-29T08:48:19.745 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-2:18.0.0-7483.g412bfade.el9.x86_64 29/33 2023-11-29T08:48:19.770 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7483.g412bfade.el9.x86_64 29/33 2023-11-29T08:48:19.770 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-29T08:48:19.770 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-11-29T08:48:19.770 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-29T08:48:19.771 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-29T08:48:19.771 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:48:39.177 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-osd-2:18.0.0-7483.g412bfade.el9.x86_64 30/33 2023-11-29T08:48:39.205 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-osd-2:18.0.0-7483.g412bfade.el9.x86_64 30/33 2023-11-29T08:48:39.205 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-29T08:48:39.205 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-11-29T08:48:39.206 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-29T08:48:39.206 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-29T08:48:39.206 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:48:40.153 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mon-2:18.0.0-7483.g412bfade.el9.x86_64 31/33 2023-11-29T08:48:40.176 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mon-2:18.0.0-7483.g412bfade.el9.x86_64 31/33 2023-11-29T08:48:40.177 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-29T08:48:40.177 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-11-29T08:48:40.177 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-29T08:48:40.177 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-29T08:48:40.177 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:48:40.386 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mds-2:18.0.0-7483.g412bfade.el9.x86_64 32/33 2023-11-29T08:48:40.410 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mds-2:18.0.0-7483.g412bfade.el9.x86_64 32/33 2023-11-29T08:48:40.410 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-29T08:48:40.410 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-11-29T08:48:40.410 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-29T08:48:40.411 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-29T08:48:40.411 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:48:40.478 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-2:18.0.0-7483.g412bfade.el9.x86_64 33/33 2023-11-29T08:48:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-2:18.0.0-7483.g412bfade.el9.x86_64 33/33 2023-11-29T08:48:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-2:18.0.0-7483.g412bfade.el9.x86_64 1/33 2023-11-29T08:48:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mds-2:18.0.0-7483.g412bfade.el9.x86_64 2/33 2023-11-29T08:48:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-2:18.0.0-7483.g412bfade.el9.x86_64 3/33 2023-11-29T08:48:50.029 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mon-2:18.0.0-7483.g412bfade.el9.x86_64 4/33 2023-11-29T08:48:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-osd-2:18.0.0-7483.g412bfade.el9.x86_64 5/33 2023-11-29T08:48:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7483.g412bfade.el9. 6/33 2023-11-29T08:48:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-11-29T08:48:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-11-29T08:48:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2023-11-29T08:48:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2023-11-29T08:48:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2023-11-29T08:48:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2023-11-29T08:48:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2023-11-29T08:48:50.030 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2023-11-29T08:48:50.031 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 15/33 2023-11-29T08:48:50.031 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 16/33 2023-11-29T08:48:50.031 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-11-29T08:48:50.031 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-11-29T08:48:50.031 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-11-29T08:48:50.031 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-11-29T08:48:50.031 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-11-29T08:48:50.031 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-11-29T08:48:50.031 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-11-29T08:48:50.032 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-11-29T08:48:50.032 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-11-29T08:48:50.032 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-11-29T08:48:50.032 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-11-29T08:48:50.033 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-11-29T08:48:50.033 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-11-29T08:48:50.033 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-11-29T08:48:50.033 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-11-29T08:48:50.033 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-11-29T08:48:50.797 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-11-29T08:48:50.797 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:48:50.797 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-11-29T08:48:50.797 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:48:50.798 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:48:50.798 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:48:50.798 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core-2:18.0.0-7483.g412bfade.el9.noarch 2023-11-29T08:48:50.798 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:48:50.798 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:48:50.798 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-11-29T08:48:50.798 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-11-29T08:48:50.798 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-11-29T08:48:50.798 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-11-29T08:48:50.798 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-11-29T08:48:50.798 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-11-29T08:48:50.799 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-11-29T08:48:50.799 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-11-29T08:48:50.799 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-11-29T08:48:50.799 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-11-29T08:48:50.799 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-11-29T08:48:50.799 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako-1.1.4-6.el9.noarch 2023-11-29T08:48:50.799 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-11-29T08:48:50.799 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-11-29T08:48:50.799 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-11-29T08:48:50.799 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply-3.11-14.el9.noarch 2023-11-29T08:48:50.799 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend-3.1.0-2.el9.noarch 2023-11-29T08:48:50.799 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-11-29T08:48:50.800 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser-2.20-6.el9.noarch 2023-11-29T08:48:50.800 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz-2021.1-5.el9.noarch 2023-11-29T08:48:50.800 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-2.25.1-7.el9.noarch 2023-11-29T08:48:50.800 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-11-29T08:48:50.800 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml-0.10.2-6.el9.noarch 2023-11-29T08:48:50.800 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-11-29T08:48:50.800 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob-1.8.7-6.el9.noarch 2023-11-29T08:48:50.800 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-11-29T08:48:50.800 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-11-29T08:48:50.800 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:48:50.800 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:48:51.021 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-base 2023-11-29T08:48:51.630 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:57 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:48:51.727 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-base-2:18.0.0-7483.g412bfade.el9.x86_64 is already installed. 2023-11-29T08:48:51.780 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:48:51.781 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-11-29T08:48:51.782 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:48:51.834 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install cephadm 2023-11-29T08:48:52.437 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:58 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:48:52.588 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:48:52.589 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:48:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-11-29T08:48:52.589 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:48:52.589 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-11-29T08:48:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm noarch 2:18.0.0-7483.g412bfade.el9 ceph-noarch 774 k 2023-11-29T08:48:52.589 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:48:52.589 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-11-29T08:48:52.589 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:48:52.590 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-11-29T08:48:52.590 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:48:52.590 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 774 k 2023-11-29T08:48:52.590 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 788 k 2023-11-29T08:48:52.590 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-11-29T08:48:54.208 INFO:teuthology.orchestra.run.smithi031.stdout:cephadm-18.0.0-7483.g412bfade.el9.noarch.rpm 478 kB/s | 774 kB 00:01 2023-11-29T08:48:54.209 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-11-29T08:48:54.209 INFO:teuthology.orchestra.run.smithi031.stdout:Total 478 kB/s | 774 kB 00:01 2023-11-29T08:48:54.209 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-11-29T08:48:54.216 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-11-29T08:48:54.216 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-11-29T08:48:54.223 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-11-29T08:48:54.223 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-11-29T08:48:54.302 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-11-29T08:48:54.731 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:18.0.0-7483.g412bfade.el9.noarch 1/1 2023-11-29T08:48:54.824 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephadm-2:18.0.0-7483.g412bfade.el9.noarch 1/1 2023-11-29T08:48:55.177 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:18.0.0-7483.g412bfade.el9.noarch 1/1 2023-11-29T08:48:55.464 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephadm-2:18.0.0-7483.g412bfade.el9.noarch 1/1 2023-11-29T08:48:55.464 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:48:55.464 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-11-29T08:48:55.464 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm-2:18.0.0-7483.g412bfade.el9.noarch 2023-11-29T08:48:55.464 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:48:55.465 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:48:55.657 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-immutable-object-cache 2023-11-29T08:48:56.265 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:02 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:48:56.416 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:48:56.417 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:48:56.417 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2023-11-29T08:48:56.417 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:48:56.417 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-11-29T08:48:56.417 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7483.g412bfade.el9 ceph 143 k 2023-11-29T08:48:56.417 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:48:56.417 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-11-29T08:48:56.417 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:48:56.417 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-11-29T08:48:56.417 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:48:56.418 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 143 k 2023-11-29T08:48:56.418 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 437 k 2023-11-29T08:48:56.418 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-11-29T08:48:57.254 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-immutable-object-cache-18.0.0-7483.g412bfa 171 kB/s | 143 kB 00:00 2023-11-29T08:48:57.255 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-11-29T08:48:57.255 INFO:teuthology.orchestra.run.smithi031.stdout:Total 171 kB/s | 143 kB 00:00 2023-11-29T08:48:57.255 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-11-29T08:48:57.264 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-11-29T08:48:57.265 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-11-29T08:48:57.324 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-11-29T08:48:57.324 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-11-29T08:48:57.469 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-11-29T08:48:57.538 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7483.g412bfade. 1/1 2023-11-29T08:48:57.563 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7483.g412bfade. 1/1 2023-11-29T08:48:57.563 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-29T08:48:57.564 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-11-29T08:48:57.564 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:48:58.604 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7483.g412bfade. 1/1 2023-11-29T08:48:58.604 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:48:58.604 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-11-29T08:48:58.604 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:48:58.604 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:48:58.604 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:48:58.783 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr 2023-11-29T08:48:59.392 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:05 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:48:59.491 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-mgr-2:18.0.0-7483.g412bfade.el9.x86_64 is already installed. 2023-11-29T08:48:59.543 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:48:59.544 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-11-29T08:48:59.545 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:48:59.604 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-dashboard 2023-11-29T08:49:00.209 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:06 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:49:00.362 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:49:00.362 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:49:00.363 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-11-29T08:49:00.363 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:49:00.363 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-11-29T08:49:00.363 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7483.g412bfade.el9 ceph-noarch 1.8 M 2023-11-29T08:49:00.363 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-11-29T08:49:00.363 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7483.g412bfade.el9 ceph-noarch 28 k 2023-11-29T08:49:00.363 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7483.g412bfade.el9 ceph-noarch 15 k 2023-11-29T08:49:00.363 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-11-29T08:49:00.363 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-11-29T08:49:00.363 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-11-29T08:49:00.364 INFO:teuthology.orchestra.run.smithi031.stdout:Installing weak dependencies: 2023-11-29T08:49:00.364 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-11-29T08:49:00.364 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:00.364 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-11-29T08:49:00.364 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:49:00.364 INFO:teuthology.orchestra.run.smithi031.stdout:Install 7 Packages 2023-11-29T08:49:00.364 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:00.365 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 2.1 M 2023-11-29T08:49:00.365 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 8.7 M 2023-11-29T08:49:00.365 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-11-29T08:49:00.982 INFO:teuthology.orchestra.run.smithi031.stdout:(1/7): ceph-prometheus-alerts-18.0.0-7483.g412b 33 kB/s | 15 kB 00:00 2023-11-29T08:49:01.049 INFO:teuthology.orchestra.run.smithi031.stdout:(2/7): ceph-grafana-dashboards-18.0.0-7483.g412 54 kB/s | 28 kB 00:00 2023-11-29T08:49:01.074 INFO:teuthology.orchestra.run.smithi031.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 98 kB/s | 9.0 kB 00:00 2023-11-29T08:49:01.116 INFO:teuthology.orchestra.run.smithi031.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 609 kB/s | 41 kB 00:00 2023-11-29T08:49:01.174 INFO:teuthology.orchestra.run.smithi031.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 308 kB/s | 31 kB 00:00 2023-11-29T08:49:01.225 INFO:teuthology.orchestra.run.smithi031.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.7 MB/s | 188 kB 00:00 2023-11-29T08:49:02.508 INFO:teuthology.orchestra.run.smithi031.stdout:(7/7): ceph-mgr-dashboard-18.0.0-7483.g412bfade 908 kB/s | 1.8 MB 00:01 2023-11-29T08:49:02.510 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-11-29T08:49:02.510 INFO:teuthology.orchestra.run.smithi031.stdout:Total 984 kB/s | 2.1 MB 00:02 2023-11-29T08:49:02.511 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-11-29T08:49:02.521 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-11-29T08:49:02.521 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-11-29T08:49:02.594 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-11-29T08:49:02.595 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-11-29T08:49:02.902 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-11-29T08:49:02.967 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-11-29T08:49:03.002 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-11-29T08:49:03.057 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-11-29T08:49:03.088 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-11-29T08:49:03.119 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7483.g412bfade.el9.n 5/7 2023-11-29T08:49:03.363 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7483.g412bfade.el9. 6/7 2023-11-29T08:49:03.506 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7483.g412bfade.el9.noarc 7/7 2023-11-29T08:49:03.959 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7483.g412bfade.el9.noarc 7/7 2023-11-29T08:49:03.959 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7483.g412bfade.el9. 1/7 2023-11-29T08:49:03.959 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7483.g412bfade.el9.noarc 2/7 2023-11-29T08:49:03.959 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7483.g412bfade.el9.n 3/7 2023-11-29T08:49:03.959 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-11-29T08:49:03.960 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-11-29T08:49:03.960 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-11-29T08:49:04.280 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-11-29T08:49:04.280 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:04.280 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-11-29T08:49:04.280 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards-2:18.0.0-7483.g412bfade.el9.noarch 2023-11-29T08:49:04.280 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard-2:18.0.0-7483.g412bfade.el9.noarch 2023-11-29T08:49:04.280 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts-2:18.0.0-7483.g412bfade.el9.noarch 2023-11-29T08:49:04.281 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-11-29T08:49:04.281 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-11-29T08:49:04.281 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-11-29T08:49:04.281 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes-2.5.1-5.el9.noarch 2023-11-29T08:49:04.281 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:04.281 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:49:04.476 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-29T08:49:05.080 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:11 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:49:05.234 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:49:05.236 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2023-11-29T08:49:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-11-29T08:49:05.236 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2023-11-29T08:49:05.236 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-11-29T08:49:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7483.g412bfade.el9 ceph-noarch 7.4 M 2023-11-29T08:49:05.236 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-11-29T08:49:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-11-29T08:49:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-11-29T08:49:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-11-29T08:49:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-11-29T08:49:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-11-29T08:49:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-11-29T08:49:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-11-29T08:49:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-11-29T08:49:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-11-29T08:49:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-11-29T08:49:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-11-29T08:49:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:05.237 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-11-29T08:49:05.237 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2023-11-29T08:49:05.237 INFO:teuthology.orchestra.run.smithi031.stdout:Install 12 Packages 2023-11-29T08:49:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:05.237 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 37 M 2023-11-29T08:49:05.237 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 209 M 2023-11-29T08:49:05.238 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-11-29T08:49:05.564 INFO:teuthology.orchestra.run.smithi031.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 237 kB/s | 32 kB 00:00 2023-11-29T08:49:05.589 INFO:teuthology.orchestra.run.smithi031.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 690 kB/s | 17 kB 00:00 2023-11-29T08:49:05.623 INFO:teuthology.orchestra.run.smithi031.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-11-29T08:49:05.824 INFO:teuthology.orchestra.run.smithi031.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 24 MB/s | 4.7 MB 00:00 2023-11-29T08:49:05.982 INFO:teuthology.orchestra.run.smithi031.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2023-11-29T08:49:06.024 INFO:teuthology.orchestra.run.smithi031.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.2 MB/s | 217 kB 00:00 2023-11-29T08:49:06.091 INFO:teuthology.orchestra.run.smithi031.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.5 MB/s | 3.0 MB 00:00 2023-11-29T08:49:06.266 INFO:teuthology.orchestra.run.smithi031.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2023-11-29T08:49:06.425 INFO:teuthology.orchestra.run.smithi031.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2023-11-29T08:49:06.450 INFO:teuthology.orchestra.run.smithi031.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2023-11-29T08:49:06.667 INFO:teuthology.orchestra.run.smithi031.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2023-11-29T08:49:20.608 INFO:teuthology.orchestra.run.smithi031.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-7 500 kB/s | 7.4 MB 00:15 2023-11-29T08:49:20.610 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-11-29T08:49:20.611 INFO:teuthology.orchestra.run.smithi031.stdout:Total 2.4 MB/s | 37 MB 00:15 2023-11-29T08:49:20.625 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-11-29T08:49:20.653 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-11-29T08:49:20.653 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-11-29T08:49:21.094 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-11-29T08:49:21.095 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-11-29T08:49:21.941 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-11-29T08:49:22.018 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-11-29T08:49:22.053 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-11-29T08:49:22.153 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-11-29T08:49:22.198 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-11-29T08:49:22.382 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-29T08:49:22.413 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-29T08:49:22.497 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-11-29T08:49:23.193 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-11-29T08:49:23.330 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-11-29T08:49:24.648 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-11-29T08:49:25.722 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-11-29T08:49:25.753 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7483.g412bf 12/12 2023-11-29T08:49:28.416 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7483.g412bf 12/12 2023-11-29T08:49:28.417 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7483.g412bf 1/12 2023-11-29T08:49:28.417 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-11-29T08:49:28.417 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-11-29T08:49:28.417 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-11-29T08:49:28.417 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-29T08:49:28.417 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-29T08:49:28.417 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-11-29T08:49:28.417 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-11-29T08:49:28.417 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-11-29T08:49:28.417 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-11-29T08:49:28.418 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-11-29T08:49:28.851 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-11-29T08:49:28.851 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:28.851 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-11-29T08:49:28.852 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7483.g412bfade.el9.noarch 2023-11-29T08:49:28.852 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-11-29T08:49:28.852 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-11-29T08:49:28.852 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-11-29T08:49:28.852 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-11-29T08:49:28.852 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-11-29T08:49:28.852 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-0.3.21-2.el9.x86_64 2023-11-29T08:49:28.852 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-11-29T08:49:28.852 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-11-29T08:49:28.852 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-11-29T08:49:28.852 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-11-29T08:49:28.853 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-11-29T08:49:28.853 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:28.853 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:49:29.240 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-rook 2023-11-29T08:49:29.853 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:35 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:49:30.006 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:49:30.007 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2023-11-29T08:49:30.007 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-11-29T08:49:30.007 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2023-11-29T08:49:30.007 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-11-29T08:49:30.008 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook noarch 2:18.0.0-7483.g412bfade.el9 ceph-noarch 50 k 2023-11-29T08:49:30.008 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-11-29T08:49:30.008 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-11-29T08:49:30.008 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-11-29T08:49:30.008 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-11-29T08:49:30.008 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-11-29T08:49:30.008 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-11-29T08:49:30.008 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-11-29T08:49:30.008 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-11-29T08:49:30.008 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-11-29T08:49:30.008 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-11-29T08:49:30.008 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-11-29T08:49:30.009 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-11-29T08:49:30.009 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-11-29T08:49:30.009 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:30.009 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-11-29T08:49:30.009 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2023-11-29T08:49:30.009 INFO:teuthology.orchestra.run.smithi031.stdout:Install 13 Packages 2023-11-29T08:49:30.009 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:30.010 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 2.0 M 2023-11-29T08:49:30.010 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 26 M 2023-11-29T08:49:30.010 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-11-29T08:49:30.196 INFO:teuthology.orchestra.run.smithi031.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 232 kB/s | 24 kB 00:00 2023-11-29T08:49:30.246 INFO:teuthology.orchestra.run.smithi031.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 116 kB/s | 18 kB 00:00 2023-11-29T08:49:30.288 INFO:teuthology.orchestra.run.smithi031.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2023-11-29T08:49:30.313 INFO:teuthology.orchestra.run.smithi031.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-11-29T08:49:30.330 INFO:teuthology.orchestra.run.smithi031.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 5.0 MB/s | 211 kB 00:00 2023-11-29T08:49:30.364 INFO:teuthology.orchestra.run.smithi031.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 871 kB/s | 43 kB 00:00 2023-11-29T08:49:30.640 INFO:teuthology.orchestra.run.smithi031.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 104 kB/s | 32 kB 00:00 2023-11-29T08:49:30.665 INFO:teuthology.orchestra.run.smithi031.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 50 kB/s | 15 kB 00:00 2023-11-29T08:49:30.690 INFO:teuthology.orchestra.run.smithi031.stdout:(9/13): ceph-mgr-rook-18.0.0-7483.g412bfade.el9 84 kB/s | 50 kB 00:00 2023-11-29T08:49:30.857 INFO:teuthology.orchestra.run.smithi031.stdout:(10/13): python3-google-auth-2.23.4-1.el9.noarc 910 kB/s | 197 kB 00:00 2023-11-29T08:49:30.957 INFO:teuthology.orchestra.run.smithi031.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 898 kB/s | 90 kB 00:00 2023-11-29T08:49:30.993 INFO:teuthology.orchestra.run.smithi031.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 197 kB/s | 59 kB 00:00 2023-11-29T08:49:31.032 INFO:teuthology.orchestra.run.smithi031.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.8 MB/s | 1.0 MB 00:00 2023-11-29T08:49:31.037 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-11-29T08:49:31.038 INFO:teuthology.orchestra.run.smithi031.stdout:Total 2.0 MB/s | 2.0 MB 00:01 2023-11-29T08:49:31.038 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-11-29T08:49:31.052 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-11-29T08:49:31.052 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-11-29T08:49:31.156 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-11-29T08:49:31.158 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-11-29T08:49:31.706 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-11-29T08:49:31.879 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-11-29T08:49:31.935 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-11-29T08:49:31.991 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-11-29T08:49:32.027 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-11-29T08:49:32.069 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-11-29T08:49:32.164 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-11-29T08:49:32.291 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-11-29T08:49:32.354 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-11-29T08:49:33.119 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-11-29T08:49:33.166 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-11-29T08:49:33.200 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-11-29T08:49:33.248 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-11-29T08:49:33.278 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-rook-2:18.0.0-7483.g412bfade.el9.noarch 13/13 2023-11-29T08:49:34.051 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7483.g412bfade.el9.noarch 13/13 2023-11-29T08:49:34.051 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7483.g412bfade.el9.noarch 1/13 2023-11-29T08:49:34.051 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-11-29T08:49:34.051 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-11-29T08:49:34.051 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-11-29T08:49:34.051 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-11-29T08:49:34.051 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-11-29T08:49:34.051 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-11-29T08:49:34.051 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-11-29T08:49:34.051 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-11-29T08:49:34.051 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-11-29T08:49:34.051 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-11-29T08:49:34.051 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-11-29T08:49:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-11-29T08:49:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:34.431 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-11-29T08:49:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook-2:18.0.0-7483.g412bfade.el9.noarch 2023-11-29T08:49:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-11-29T08:49:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-11-29T08:49:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-11-29T08:49:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-11-29T08:49:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-11-29T08:49:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-11-29T08:49:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-11-29T08:49:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-11-29T08:49:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-11-29T08:49:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-11-29T08:49:34.432 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa-4.9-2.el9.noarch 2023-11-29T08:49:34.432 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-11-29T08:49:34.432 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:34.432 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:49:34.675 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-cephadm 2023-11-29T08:49:35.285 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:41 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:49:35.437 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:49:35.438 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2023-11-29T08:49:35.438 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-11-29T08:49:35.438 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2023-11-29T08:49:35.438 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-11-29T08:49:35.438 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7483.g412bfade.el9 ceph-noarch 133 k 2023-11-29T08:49:35.438 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-11-29T08:49:35.439 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-11-29T08:49:35.439 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-11-29T08:49:35.439 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-11-29T08:49:35.439 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-11-29T08:49:35.439 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-11-29T08:49:35.439 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:35.439 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-11-29T08:49:35.439 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2023-11-29T08:49:35.439 INFO:teuthology.orchestra.run.smithi031.stdout:Install 6 Packages 2023-11-29T08:49:35.439 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:35.440 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 6.8 M 2023-11-29T08:49:35.440 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 33 M 2023-11-29T08:49:35.440 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-11-29T08:49:35.775 INFO:teuthology.orchestra.run.smithi031.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2023-11-29T08:49:35.883 INFO:teuthology.orchestra.run.smithi031.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2023-11-29T08:49:36.167 INFO:teuthology.orchestra.run.smithi031.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 223 kB/s | 58 kB 00:00 2023-11-29T08:49:36.225 INFO:teuthology.orchestra.run.smithi031.stdout:(4/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2023-11-29T08:49:36.250 INFO:teuthology.orchestra.run.smithi031.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 610 kB/s | 51 kB 00:00 2023-11-29T08:49:36.567 INFO:teuthology.orchestra.run.smithi031.stdout:(6/6): ceph-mgr-cephadm-18.0.0-7483.g412bfade.e 137 kB/s | 133 kB 00:00 2023-11-29T08:49:36.571 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-11-29T08:49:36.571 INFO:teuthology.orchestra.run.smithi031.stdout:Total 6.0 MB/s | 6.8 MB 00:01 2023-11-29T08:49:36.571 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-11-29T08:49:36.581 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-11-29T08:49:36.582 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-11-29T08:49:36.680 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-11-29T08:49:36.681 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-11-29T08:49:36.992 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-11-29T08:49:37.165 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-11-29T08:49:37.220 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-11-29T08:49:37.779 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-11-29T08:49:37.865 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-11-29T08:49:37.942 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-11-29T08:49:37.965 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7483.g412bfade.el9.noarch 6/6 2023-11-29T08:49:38.880 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7483.g412bfade.el9.noarch 6/6 2023-11-29T08:49:38.880 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7483.g412bfade.el9.noarch 1/6 2023-11-29T08:49:38.880 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-11-29T08:49:38.880 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-11-29T08:49:38.880 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-11-29T08:49:38.880 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-11-29T08:49:39.210 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-11-29T08:49:39.210 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:39.210 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-11-29T08:49:39.211 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm-2:18.0.0-7483.g412bfade.el9.noarch 2023-11-29T08:49:39.211 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-11-29T08:49:39.211 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel-2.9.1-2.el9.noarch 2023-11-29T08:49:39.211 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-11-29T08:49:39.211 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-11-29T08:49:39.211 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-11-29T08:49:39.211 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:39.211 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:49:39.452 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-fuse 2023-11-29T08:49:40.057 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:46 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:49:40.208 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:49:40.208 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:49:40.208 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-11-29T08:49:40.208 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:49:40.208 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-11-29T08:49:40.208 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse x86_64 2:18.0.0-7483.g412bfade.el9 ceph 833 k 2023-11-29T08:49:40.208 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-11-29T08:49:40.208 INFO:teuthology.orchestra.run.smithi031.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-11-29T08:49:40.209 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:40.209 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-11-29T08:49:40.209 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:49:40.209 INFO:teuthology.orchestra.run.smithi031.stdout:Install 2 Packages 2023-11-29T08:49:40.209 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:40.209 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 911 k 2023-11-29T08:49:40.209 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 3.0 M 2023-11-29T08:49:40.209 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-11-29T08:49:40.327 INFO:teuthology.orchestra.run.smithi031.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 670 kB/s | 79 kB 00:00 2023-11-29T08:49:43.302 INFO:teuthology.orchestra.run.smithi031.stdout:(2/2): ceph-fuse-18.0.0-7483.g412bfade.el9.x86_ 269 kB/s | 833 kB 00:03 2023-11-29T08:49:43.303 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-11-29T08:49:43.304 INFO:teuthology.orchestra.run.smithi031.stdout:Total 294 kB/s | 911 kB 00:03 2023-11-29T08:49:43.304 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-11-29T08:49:43.330 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-11-29T08:49:43.330 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-11-29T08:49:43.418 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-11-29T08:49:43.418 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-11-29T08:49:43.610 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-11-29T08:49:43.781 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-11-29T08:49:43.853 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-fuse-2:18.0.0-7483.g412bfade.el9.x86_64 2/2 2023-11-29T08:49:44.537 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7483.g412bfade.el9.x86_64 2/2 2023-11-29T08:49:44.537 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-fuse-2:18.0.0-7483.g412bfade.el9.x86_64 1/2 2023-11-29T08:49:44.869 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-11-29T08:49:44.870 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:44.870 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-11-29T08:49:44.870 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse-2:18.0.0-7483.g412bfade.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-11-29T08:49:44.870 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:44.870 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:49:45.133 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-volume 2023-11-29T08:49:45.750 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:51 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:49:45.902 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:49:45.903 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:49:45.903 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-11-29T08:49:45.903 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:49:45.903 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-11-29T08:49:45.903 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume noarch 2:18.0.0-7483.g412bfade.el9 ceph-noarch 262 k 2023-11-29T08:49:45.903 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:45.903 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-11-29T08:49:45.903 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:49:45.903 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-11-29T08:49:45.904 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:45.904 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 262 k 2023-11-29T08:49:45.904 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 1.2 M 2023-11-29T08:49:45.904 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-11-29T08:49:46.772 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-volume-18.0.0-7483.g412bfade.el9.noarch.rp 302 kB/s | 262 kB 00:00 2023-11-29T08:49:46.773 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-11-29T08:49:46.773 INFO:teuthology.orchestra.run.smithi031.stdout:Total 301 kB/s | 262 kB 00:00 2023-11-29T08:49:46.773 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-11-29T08:49:46.783 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-11-29T08:49:46.783 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-11-29T08:49:46.829 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-11-29T08:49:46.829 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-11-29T08:49:47.061 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-11-29T08:49:47.133 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-volume-2:18.0.0-7483.g412bfade.el9.noarch 1/1 2023-11-29T08:49:47.149 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-volume-2:18.0.0-7483.g412bfade.el9.noarch 1/1 2023-11-29T08:49:47.149 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-29T08:49:47.149 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-11-29T08:49:47.149 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:48.251 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-volume-2:18.0.0-7483.g412bfade.el9.noarch 1/1 2023-11-29T08:49:48.252 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:48.252 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-11-29T08:49:48.252 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume-2:18.0.0-7483.g412bfade.el9.noarch 2023-11-29T08:49:48.252 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:48.252 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:49:48.543 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librados-devel 2023-11-29T08:49:49.163 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:55 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:49:49.314 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:49:49.315 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:49:49.315 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2023-11-29T08:49:49.315 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:49:49.315 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-11-29T08:49:49.315 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel x86_64 2:18.0.0-7483.g412bfade.el9 ceph 127 k 2023-11-29T08:49:49.315 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:49.315 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-11-29T08:49:49.315 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:49:49.315 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-11-29T08:49:49.315 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:49.316 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 127 k 2023-11-29T08:49:49.316 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 456 k 2023-11-29T08:49:49.316 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-11-29T08:49:49.935 INFO:teuthology.orchestra.run.smithi031.stdout:librados-devel-18.0.0-7483.g412bfade.el9.x86_64 205 kB/s | 127 kB 00:00 2023-11-29T08:49:49.935 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-11-29T08:49:49.936 INFO:teuthology.orchestra.run.smithi031.stdout:Total 204 kB/s | 127 kB 00:00 2023-11-29T08:49:49.936 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-11-29T08:49:49.942 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-11-29T08:49:49.942 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-11-29T08:49:50.003 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-11-29T08:49:50.003 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-11-29T08:49:50.158 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-11-29T08:49:50.270 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librados-devel-2:18.0.0-7483.g412bfade.el9.x86_64 1/1 2023-11-29T08:49:50.699 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados-devel-2:18.0.0-7483.g412bfade.el9.x86_64 1/1 2023-11-29T08:49:51.001 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados-devel-2:18.0.0-7483.g412bfade.el9.x86_64 1/1 2023-11-29T08:49:51.001 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:51.001 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-11-29T08:49:51.001 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:49:51.001 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:51.001 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:49:51.223 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install libcephfs2 2023-11-29T08:49:51.833 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:57 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:49:51.931 INFO:teuthology.orchestra.run.smithi031.stdout:Package libcephfs2-2:18.0.0-7483.g412bfade.el9.x86_64 is already installed. 2023-11-29T08:49:51.984 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:49:51.986 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-11-29T08:49:51.986 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:49:52.048 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install libcephfs-devel 2023-11-29T08:49:52.657 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:58 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:49:52.807 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:49:52.808 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:49:52.808 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2023-11-29T08:49:52.808 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:49:52.808 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-11-29T08:49:52.808 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel x86_64 2:18.0.0-7483.g412bfade.el9 ceph 31 k 2023-11-29T08:49:52.808 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:52.808 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-11-29T08:49:52.808 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:49:52.809 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-11-29T08:49:52.809 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:52.809 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 31 k 2023-11-29T08:49:52.809 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 139 k 2023-11-29T08:49:52.809 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-11-29T08:49:53.330 INFO:teuthology.orchestra.run.smithi031.stdout:libcephfs-devel-18.0.0-7483.g412bfade.el9.x86_6 60 kB/s | 31 kB 00:00 2023-11-29T08:49:53.331 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-11-29T08:49:53.331 INFO:teuthology.orchestra.run.smithi031.stdout:Total 60 kB/s | 31 kB 00:00 2023-11-29T08:49:53.331 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-11-29T08:49:53.334 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-11-29T08:49:53.335 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-11-29T08:49:53.360 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-11-29T08:49:53.361 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-11-29T08:49:53.452 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-11-29T08:49:53.555 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs-devel-2:18.0.0-7483.g412bfade.el9.x86_64 1/1 2023-11-29T08:49:53.957 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7483.g412bfade.el9.x86_64 1/1 2023-11-29T08:49:54.235 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs-devel-2:18.0.0-7483.g412bfade.el9.x86_64 1/1 2023-11-29T08:49:54.236 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:54.236 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-11-29T08:49:54.236 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:49:54.236 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:49:54.236 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:49:54.405 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librados2 2023-11-29T08:49:55.015 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:01 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:49:55.113 INFO:teuthology.orchestra.run.smithi031.stdout:Package librados2-2:18.0.0-7483.g412bfade.el9.x86_64 is already installed. 2023-11-29T08:49:55.166 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:49:55.168 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-11-29T08:49:55.168 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:49:55.228 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librbd1 2023-11-29T08:49:55.848 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:01 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:49:55.947 INFO:teuthology.orchestra.run.smithi031.stdout:Package librbd1-2:18.0.0-7483.g412bfade.el9.x86_64 is already installed. 2023-11-29T08:49:56.000 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:49:56.001 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-11-29T08:49:56.002 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:49:56.059 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rados 2023-11-29T08:49:56.677 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:02 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:49:56.778 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rados-2:18.0.0-7483.g412bfade.el9.x86_64 is already installed. 2023-11-29T08:49:56.833 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:49:56.834 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-11-29T08:49:56.834 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:49:56.897 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rgw 2023-11-29T08:49:57.513 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:03 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:49:57.611 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rgw-2:18.0.0-7483.g412bfade.el9.x86_64 is already installed. 2023-11-29T08:49:57.664 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:49:57.665 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-11-29T08:49:57.665 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:49:57.719 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-cephfs 2023-11-29T08:49:58.328 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:04 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:49:58.426 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-cephfs-2:18.0.0-7483.g412bfade.el9.x86_64 is already installed. 2023-11-29T08:49:58.478 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:49:58.480 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-11-29T08:49:58.480 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:49:58.539 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rbd 2023-11-29T08:49:59.147 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:05 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:49:59.244 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rbd-2:18.0.0-7483.g412bfade.el9.x86_64 is already installed. 2023-11-29T08:49:59.298 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:49:59.299 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-11-29T08:49:59.299 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:49:59.352 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-fuse 2023-11-29T08:49:59.957 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:05 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:50:00.109 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:50:00.110 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:50:00.110 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2023-11-29T08:50:00.110 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:50:00.110 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-11-29T08:50:00.110 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse x86_64 2:18.0.0-7483.g412bfade.el9 ceph 86 k 2023-11-29T08:50:00.111 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:00.111 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-11-29T08:50:00.111 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:50:00.111 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-11-29T08:50:00.111 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:00.111 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 86 k 2023-11-29T08:50:00.111 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 256 k 2023-11-29T08:50:00.111 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-11-29T08:50:00.800 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-fuse-18.0.0-7483.g412bfade.el9.x86_64.rpm 125 kB/s | 86 kB 00:00 2023-11-29T08:50:00.801 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-11-29T08:50:00.801 INFO:teuthology.orchestra.run.smithi031.stdout:Total 124 kB/s | 86 kB 00:00 2023-11-29T08:50:00.801 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-11-29T08:50:00.809 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-11-29T08:50:00.809 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-11-29T08:50:00.870 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-11-29T08:50:00.870 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-11-29T08:50:01.017 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-11-29T08:50:01.117 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-fuse-2:18.0.0-7483.g412bfade.el9.x86_64 1/1 2023-11-29T08:50:01.553 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7483.g412bfade.el9.x86_64 1/1 2023-11-29T08:50:01.888 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-fuse-2:18.0.0-7483.g412bfade.el9.x86_64 1/1 2023-11-29T08:50:01.888 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:01.888 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-11-29T08:50:01.888 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:50:01.888 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:01.888 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:50:02.122 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-mirror 2023-11-29T08:50:02.732 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:08 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:50:02.885 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:50:02.886 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:50:02.886 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2023-11-29T08:50:02.886 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:50:02.886 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-11-29T08:50:02.886 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror x86_64 2:18.0.0-7483.g412bfade.el9 ceph 3.0 M 2023-11-29T08:50:02.886 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:02.886 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-11-29T08:50:02.886 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:50:02.886 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-11-29T08:50:02.887 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:02.887 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 3.0 M 2023-11-29T08:50:02.887 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 13 M 2023-11-29T08:50:02.887 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-11-29T08:50:08.552 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-mirror-18.0.0-7483.g412bfade.el9.x86_64.rpm 540 kB/s | 3.0 MB 00:05 2023-11-29T08:50:08.552 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-11-29T08:50:08.553 INFO:teuthology.orchestra.run.smithi031.stdout:Total 540 kB/s | 3.0 MB 00:05 2023-11-29T08:50:08.553 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-11-29T08:50:08.565 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-11-29T08:50:08.565 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-11-29T08:50:08.646 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-11-29T08:50:08.646 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-11-29T08:50:09.101 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-11-29T08:50:09.160 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-mirror-2:18.0.0-7483.g412bfade.el9.x86_64 1/1 2023-11-29T08:50:09.187 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7483.g412bfade.el9.x86_64 1/1 2023-11-29T08:50:09.188 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-29T08:50:09.188 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-11-29T08:50:09.188 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-29T08:50:09.188 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-29T08:50:09.188 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:10.436 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-mirror-2:18.0.0-7483.g412bfade.el9.x86_64 1/1 2023-11-29T08:50:10.436 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:10.436 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-11-29T08:50:10.436 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:50:10.436 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:10.436 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:50:10.664 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-nbd 2023-11-29T08:50:11.276 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:17 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:50:11.428 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:50:11.429 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:50:11.429 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2023-11-29T08:50:11.429 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:50:11.429 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-11-29T08:50:11.429 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd x86_64 2:18.0.0-7483.g412bfade.el9 ceph 169 k 2023-11-29T08:50:11.429 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:11.429 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-11-29T08:50:11.429 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:50:11.429 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-11-29T08:50:11.430 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:11.430 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 169 k 2023-11-29T08:50:11.430 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 536 k 2023-11-29T08:50:11.430 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-11-29T08:50:12.221 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-nbd-18.0.0-7483.g412bfade.el9.x86_64.rpm 213 kB/s | 169 kB 00:00 2023-11-29T08:50:12.221 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-11-29T08:50:12.222 INFO:teuthology.orchestra.run.smithi031.stdout:Total 213 kB/s | 169 kB 00:00 2023-11-29T08:50:12.222 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-11-29T08:50:12.231 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-11-29T08:50:12.231 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-11-29T08:50:12.293 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-11-29T08:50:12.293 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-11-29T08:50:12.450 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-11-29T08:50:12.546 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-nbd-2:18.0.0-7483.g412bfade.el9.x86_64 1/1 2023-11-29T08:50:12.977 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7483.g412bfade.el9.x86_64 1/1 2023-11-29T08:50:13.268 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-nbd-2:18.0.0-7483.g412bfade.el9.x86_64 1/1 2023-11-29T08:50:13.268 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:13.268 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-11-29T08:50:13.268 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:50:13.268 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:13.268 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:50:13.465 DEBUG:teuthology.parallel:result is None 2023-11-29T08:50:13.466 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=412bfade5c44735b71d7c14eb55649269cd32a35 2023-11-29T08:50:13.684 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-29T08:50:13.706 INFO:teuthology.orchestra.run.smithi031.stdout:18.0.0-7483.g412bfade.el9 2023-11-29T08:50:13.706 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7483.g412bfade.el9 2023-11-29T08:50:13.706 INFO:teuthology.task.install:The correct ceph version 18.0.0-7483.g412bfade is installed. 2023-11-29T08:50:13.707 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-29T08:50:13.707 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-11-29T08:50:13.707 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-29T08:50:13.778 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-29T08:50:13.778 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-11-29T08:50:13.778 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/daemon-helper 2023-11-29T08:50:13.849 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-29T08:50:13.919 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-29T08:50:13.919 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-11-29T08:50:13.919 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-29T08:50:13.989 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-29T08:50:14.060 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-29T08:50:14.060 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-11-29T08:50:14.060 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/stdin-killer 2023-11-29T08:50:14.132 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-29T08:50:14.204 INFO:teuthology.run_tasks:Running task ceph... 2023-11-29T08:50:14.269 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-29T08:50:14.269 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 777 /var/log/ceph 2023-11-29T08:50:14.300 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-29T08:50:14.300 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-29T08:50:14.371 INFO:tasks.ceph:Creating extra log directories... 2023-11-29T08:50:14.371 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-29T08:50:14.448 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-11-29T08:50:14.449 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-29T08:50:14.449 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2023-11-29_08:30:41-crimson-rados-wip-matanb-crimson-snaptrimevent-lifetime-v2-distro-crimson-smithi/7470487', 'branch': 'wip-matanb-crimson-snaptrimevent-lifetime-v2', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_write}', 'email': None, 'first_in_suite': False, 'job_id': '7470487', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2023-11-29_08:30:41-crimson-rados-wip-matanb-crimson-snaptrimevent-lifetime-v2-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-snaptrimevent-lifetime-v2'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '412bfade5c44735b71d7c14eb55649269cd32a35'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '412bfade5c44735b71d7c14eb55649269cd32a35'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '3d760f113bd9f3a2e513474d34227633cd06f1ea'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 4297, 'sha1': '412bfade5c44735b71d7c14eb55649269cd32a35', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_3d760f113bd9f3a2e513474d34227633cd06f1ea/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': '3d760f113bd9f3a2e513474d34227633cd06f1ea', 'targets': {'smithi031.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGsnONvKlQj1qDG+OX3AiyN17NuwYgVbVMLBMf+Q/SyQ5q+jg/0ZkroA3iuHPBwo1hTF8rBOTrrWA1rTL3h+JUI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '412bfade5c44735b71d7c14eb55649269cd32a35', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randwrite'], 'norandommap': True, 'op_size': [4194304], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2]}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '735a43c3d8e74286f980f1d14817fb093cd8147e', 'timestamp': '2023-11-29_08:30:41', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-11-29T08:50:14.449 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-29T08:50:14.497 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/run/ceph 2023-11-29T08:50:14.568 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-11-29T08:50:14.568 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout 2023-11-29T08:50:14.624 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-29T08:50:14.624 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1 2023-11-29T08:50:14.679 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-29T08:50:14.679 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-29T08:50:14.679 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 663 Links: 1 2023-11-29T08:50:14.679 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-29T08:50:14.679 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-11-29T08:50:14.680 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-11-29 08:50:09.435139889 +0000 2023-11-29T08:50:14.680 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-11-29 08:44:52.053711718 +0000 2023-11-29T08:50:14.680 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-11-29 08:44:52.053711718 +0000 2023-11-29T08:50:14.680 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2023-11-29 08:44:52.053711718 +0000 2023-11-29T08:50:14.680 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-29T08:50:14.748 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-11-29T08:50:14.748 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-11-29T08:50:14.748 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000196682 s, 2.6 MB/s 2023-11-29T08:50:14.750 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-29T08:50:14.807 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2 2023-11-29T08:50:14.862 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-29T08:50:14.863 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-29T08:50:14.863 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 682 Links: 1 2023-11-29T08:50:14.863 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-29T08:50:14.863 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-11-29T08:50:14.863 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-11-29 08:50:09.436139872 +0000 2023-11-29T08:50:14.863 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-11-29 08:44:52.591702813 +0000 2023-11-29T08:50:14.863 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-11-29 08:44:52.591702813 +0000 2023-11-29T08:50:14.863 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2023-11-29 08:44:52.591702813 +0000 2023-11-29T08:50:14.863 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-29T08:50:14.932 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-11-29T08:50:14.932 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-11-29T08:50:14.932 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000176342 s, 2.9 MB/s 2023-11-29T08:50:14.934 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-29T08:50:14.990 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3 2023-11-29T08:50:15.045 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-29T08:50:15.046 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-29T08:50:15.046 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 702 Links: 1 2023-11-29T08:50:15.046 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-29T08:50:15.046 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-11-29T08:50:15.046 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-11-29 08:50:09.436139872 +0000 2023-11-29T08:50:15.046 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-11-29 08:44:53.141693710 +0000 2023-11-29T08:50:15.046 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-11-29 08:44:53.141693710 +0000 2023-11-29T08:50:15.046 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2023-11-29 08:44:53.141693710 +0000 2023-11-29T08:50:15.046 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-29T08:50:15.115 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-11-29T08:50:15.115 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-11-29T08:50:15.115 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000186528 s, 2.7 MB/s 2023-11-29T08:50:15.117 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-29T08:50:15.173 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4 2023-11-29T08:50:15.229 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-29T08:50:15.229 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-29T08:50:15.229 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-11-29T08:50:15.229 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-29T08:50:15.229 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-11-29T08:50:15.229 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-11-29 08:50:09.437139856 +0000 2023-11-29T08:50:15.229 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-11-29 08:44:53.699684474 +0000 2023-11-29T08:50:15.229 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-11-29 08:44:53.699684474 +0000 2023-11-29T08:50:15.229 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2023-11-29 08:44:53.699684474 +0000 2023-11-29T08:50:15.230 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-29T08:50:15.298 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-11-29T08:50:15.298 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-11-29T08:50:15.299 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000256803 s, 2.0 MB/s 2023-11-29T08:50:15.300 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-29T08:50:15.356 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-29T08:50:15.356 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-29T08:50:15.357 INFO:tasks.ceph:Generating config... 2023-11-29T08:50:15.358 INFO:tasks.ceph:[osd] debug monc = 20 2023-11-29T08:50:15.358 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-29T08:50:15.358 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2023-11-29T08:50:15.358 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-11-29T08:50:15.358 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-29T08:50:15.358 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2023-11-29T08:50:15.358 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2023-11-29T08:50:15.359 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-11-29T08:50:15.359 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-11-29T08:50:15.359 INFO:tasks.ceph:[osd] debug filestore = 0/0 2023-11-29T08:50:15.359 INFO:tasks.ceph:[osd] debug journal = 0/0 2023-11-29T08:50:15.359 INFO:tasks.ceph:[osd] debug ms = 0/0 2023-11-29T08:50:15.359 INFO:tasks.ceph:[osd] debug osd = 0/0 2023-11-29T08:50:15.359 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-11-29T08:50:15.359 INFO:tasks.ceph:[osd] err to stderr = True 2023-11-29T08:50:15.359 INFO:tasks.ceph:[osd] log flush on exit = True 2023-11-29T08:50:15.359 INFO:tasks.ceph:[osd] log to file = False 2023-11-29T08:50:15.359 INFO:tasks.ceph:[osd] log to stderr = True 2023-11-29T08:50:15.359 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-29T08:50:15.360 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-29T08:50:15.360 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-29T08:50:15.360 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-29T08:50:15.360 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-29T08:50:15.360 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-11-29T08:50:15.360 INFO:tasks.ceph:[global] auth client required = none 2023-11-29T08:50:15.360 INFO:tasks.ceph:[global] auth cluster required = none 2023-11-29T08:50:15.360 INFO:tasks.ceph:[global] auth service required = none 2023-11-29T08:50:15.360 INFO:tasks.ceph:[global] auth supported = none 2023-11-29T08:50:15.360 INFO:tasks.ceph:[global] debug asok = 0/0 2023-11-29T08:50:15.360 INFO:tasks.ceph:[global] debug auth = 0/0 2023-11-29T08:50:15.361 INFO:tasks.ceph:[global] debug bdev = 0/0 2023-11-29T08:50:15.361 INFO:tasks.ceph:[global] debug bluefs = 0/0 2023-11-29T08:50:15.361 INFO:tasks.ceph:[global] debug bluestore = 0/0 2023-11-29T08:50:15.361 INFO:tasks.ceph:[global] debug buffer = 0/0 2023-11-29T08:50:15.361 INFO:tasks.ceph:[global] debug civetweb = 0/0 2023-11-29T08:50:15.361 INFO:tasks.ceph:[global] debug client = 0/0 2023-11-29T08:50:15.361 INFO:tasks.ceph:[global] debug compressor = 0/0 2023-11-29T08:50:15.361 INFO:tasks.ceph:[global] debug context = 0/0 2023-11-29T08:50:15.361 INFO:tasks.ceph:[global] debug crush = 0/0 2023-11-29T08:50:15.361 INFO:tasks.ceph:[global] debug crypto = 0/0 2023-11-29T08:50:15.361 INFO:tasks.ceph:[global] debug dpdk = 0/0 2023-11-29T08:50:15.361 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2023-11-29T08:50:15.361 INFO:tasks.ceph:[global] debug filer = 0/0 2023-11-29T08:50:15.362 INFO:tasks.ceph:[global] debug filestore = 0/0 2023-11-29T08:50:15.362 INFO:tasks.ceph:[global] debug finisher = 0/0 2023-11-29T08:50:15.362 INFO:tasks.ceph:[global] debug fuse = 0/0 2023-11-29T08:50:15.362 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2023-11-29T08:50:15.362 INFO:tasks.ceph:[global] debug javaclient = 0/0 2023-11-29T08:50:15.362 INFO:tasks.ceph:[global] debug journal = 0/0 2023-11-29T08:50:15.362 INFO:tasks.ceph:[global] debug journaler = 0/0 2023-11-29T08:50:15.362 INFO:tasks.ceph:[global] debug kstore = 0/0 2023-11-29T08:50:15.362 INFO:tasks.ceph:[global] debug leveldb = 0/0 2023-11-29T08:50:15.362 INFO:tasks.ceph:[global] debug lockdep = 0/0 2023-11-29T08:50:15.362 INFO:tasks.ceph:[global] debug mds = 0/0 2023-11-29T08:50:15.363 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2023-11-29T08:50:15.363 INFO:tasks.ceph:[global] debug mds locker = 0/0 2023-11-29T08:50:15.363 INFO:tasks.ceph:[global] debug mds log = 0/0 2023-11-29T08:50:15.363 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2023-11-29T08:50:15.363 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2023-11-29T08:50:15.363 INFO:tasks.ceph:[global] debug memdb = 0/0 2023-11-29T08:50:15.363 INFO:tasks.ceph:[global] debug mgr = 0/0 2023-11-29T08:50:15.363 INFO:tasks.ceph:[global] debug mgrc = 0/0 2023-11-29T08:50:15.363 INFO:tasks.ceph:[global] debug mon = 0/0 2023-11-29T08:50:15.363 INFO:tasks.ceph:[global] debug monc = 0/0 2023-11-29T08:50:15.363 INFO:tasks.ceph:[global] debug ms = 0/0 2023-11-29T08:50:15.364 INFO:tasks.ceph:[global] debug objclass = 0/0 2023-11-29T08:50:15.364 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2023-11-29T08:50:15.364 INFO:tasks.ceph:[global] debug objecter = 0/0 2023-11-29T08:50:15.364 INFO:tasks.ceph:[global] debug optracker = 0/0 2023-11-29T08:50:15.364 INFO:tasks.ceph:[global] debug osd = 0/0 2023-11-29T08:50:15.364 INFO:tasks.ceph:[global] debug paxos = 0/0 2023-11-29T08:50:15.364 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2023-11-29T08:50:15.364 INFO:tasks.ceph:[global] debug rados = 0/0 2023-11-29T08:50:15.364 INFO:tasks.ceph:[global] debug rbd = 0/0 2023-11-29T08:50:15.364 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2023-11-29T08:50:15.364 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2023-11-29T08:50:15.364 INFO:tasks.ceph:[global] debug refs = 0/0 2023-11-29T08:50:15.365 INFO:tasks.ceph:[global] debug rgw = 0/0 2023-11-29T08:50:15.365 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2023-11-29T08:50:15.365 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2023-11-29T08:50:15.365 INFO:tasks.ceph:[global] debug striper = 0/0 2023-11-29T08:50:15.365 INFO:tasks.ceph:[global] debug throttle = 0/0 2023-11-29T08:50:15.365 INFO:tasks.ceph:[global] debug timer = 0/0 2023-11-29T08:50:15.365 INFO:tasks.ceph:[global] debug tp = 0/0 2023-11-29T08:50:15.365 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2023-11-29T08:50:15.365 INFO:tasks.ceph:[global] ms client mode = crc 2023-11-29T08:50:15.365 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-11-29T08:50:15.365 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-11-29T08:50:15.365 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-11-29T08:50:15.366 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-11-29T08:50:15.366 INFO:tasks.ceph:[global] ms service mode = crc 2023-11-29T08:50:15.366 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-29T08:50:15.366 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-29T08:50:15.366 INFO:tasks.ceph:[mon] debug mon = 0/0 2023-11-29T08:50:15.366 INFO:tasks.ceph:[mon] debug ms = 0/0 2023-11-29T08:50:15.366 INFO:tasks.ceph:[mon] debug paxos = 0/0 2023-11-29T08:50:15.366 INFO:tasks.ceph:[mon] osd pool default crimson = True 2023-11-29T08:50:15.366 INFO:tasks.ceph:Setting up mon.a... 2023-11-29T08:50:15.367 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-11-29T08:50:15.479 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/ceph.keyring 2023-11-29T08:50:15.482 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-11-29T08:50:15.535 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-29T08:50:15.565 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.31')] 2023-11-29T08:50:15.566 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.31', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-11-29T08:50:15.567 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-11-29T08:50:15.567 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-29T08:50:15.620 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.31 --print /home/ubuntu/cephtest/ceph.monmap 2023-11-29T08:50:15.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:15.704+0000 7fb7a9a2ae80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:15.705 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:15.704+0000 7fb7a9a2ae80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:15.706 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:15.704+0000 7fb7a9a2ae80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:15.706 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-11-29T08:50:15.706 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: generated fsid 5ce98019-9fb4-4498-915a-520cb621b53d 2023-11-29T08:50:15.706 INFO:teuthology.orchestra.run.smithi031.stdout:setting min_mon_release = pacific 2023-11-29T08:50:15.707 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 0 2023-11-29T08:50:15.707 INFO:teuthology.orchestra.run.smithi031.stdout:fsid 5ce98019-9fb4-4498-915a-520cb621b53d 2023-11-29T08:50:15.707 INFO:teuthology.orchestra.run.smithi031.stdout:last_changed 2023-11-29T08:50:15.705861+0000 2023-11-29T08:50:15.707 INFO:teuthology.orchestra.run.smithi031.stdout:created 2023-11-29T08:50:15.705861+0000 2023-11-29T08:50:15.707 INFO:teuthology.orchestra.run.smithi031.stdout:min_mon_release 16 (pacific) 2023-11-29T08:50:15.707 INFO:teuthology.orchestra.run.smithi031.stdout:election_strategy: 1 2023-11-29T08:50:15.707 INFO:teuthology.orchestra.run.smithi031.stdout:0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2023-11-29T08:50:15.707 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-11-29T08:50:15.708 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-29T08:50:15.723 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 5ce98019-9fb4-4498-915a-520cb621b53d... 2023-11-29T08:50:15.724 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-29T08:50:15.824 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2023-11-29T08:50:15.824 INFO:teuthology.orchestra.run.smithi031.stdout: chdir = "" 2023-11-29T08:50:15.824 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-29T08:50:15.824 INFO:teuthology.orchestra.run.smithi031.stdout: auth supported = none 2023-11-29T08:50:15.824 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.824 INFO:teuthology.orchestra.run.smithi031.stdout: filestore xattr use omap = true 2023-11-29T08:50:15.824 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.824 INFO:teuthology.orchestra.run.smithi031.stdout: mon clock drift allowed = 1.000 2023-11-29T08:50:15.824 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.825 INFO:teuthology.orchestra.run.smithi031.stdout: osd crush chooseleaf type = 0 2023-11-29T08:50:15.825 INFO:teuthology.orchestra.run.smithi031.stdout: auth debug = true 2023-11-29T08:50:15.825 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.825 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on old message = true 2023-11-29T08:50:15.825 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on bug = true 2023-11-29T08:50:15.825 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.825 INFO:teuthology.orchestra.run.smithi031.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-29T08:50:15.825 INFO:teuthology.orchestra.run.smithi031.stdout: mon pg warn max object skew = 0 2023-11-29T08:50:15.825 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.825 INFO:teuthology.orchestra.run.smithi031.stdout: # disable pg_autoscaler by default for new pools 2023-11-29T08:50:15.825 INFO:teuthology.orchestra.run.smithi031.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-29T08:50:15.825 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.825 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default size = 2 2023-11-29T08:50:15.825 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.825 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow primary affinity = true 2023-11-29T08:50:15.826 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow pg remap = true 2023-11-29T08:50:15.826 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on legacy crush tunables = false 2023-11-29T08:50:15.826 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on crush straw calc version zero = false 2023-11-29T08:50:15.826 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on no sortbitwise = false 2023-11-29T08:50:15.826 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on osd down out interval zero = false 2023-11-29T08:50:15.826 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on too few osds = false 2023-11-29T08:50:15.826 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-29T08:50:15.826 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-29T08:50:15.826 INFO:teuthology.orchestra.run.smithi031.stdout: mon_allow_pool_size_one = true 2023-11-29T08:50:15.826 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.826 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-29T08:50:15.827 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.827 INFO:teuthology.orchestra.run.smithi031.stdout: osd default data pool replay window = 5 2023-11-29T08:50:15.827 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.827 INFO:teuthology.orchestra.run.smithi031.stdout: mon allow pool delete = true 2023-11-29T08:50:15.827 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.827 INFO:teuthology.orchestra.run.smithi031.stdout: mon cluster log file level = debug 2023-11-29T08:50:15.827 INFO:teuthology.orchestra.run.smithi031.stdout: debug asserts on shutdown = true 2023-11-29T08:50:15.827 INFO:teuthology.orchestra.run.smithi031.stdout: mon health detail to clog = false 2023-11-29T08:50:15.827 INFO:teuthology.orchestra.run.smithi031.stdout: mon host = 172.21.15.31 2023-11-29T08:50:15.827 INFO:teuthology.orchestra.run.smithi031.stdout: auth client required = none 2023-11-29T08:50:15.827 INFO:teuthology.orchestra.run.smithi031.stdout: auth cluster required = none 2023-11-29T08:50:15.827 INFO:teuthology.orchestra.run.smithi031.stdout: auth service required = none 2023-11-29T08:50:15.827 INFO:teuthology.orchestra.run.smithi031.stdout: debug asok = 0/0 2023-11-29T08:50:15.827 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 0/0 2023-11-29T08:50:15.828 INFO:teuthology.orchestra.run.smithi031.stdout: debug bdev = 0/0 2023-11-29T08:50:15.828 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluefs = 0/0 2023-11-29T08:50:15.828 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluestore = 0/0 2023-11-29T08:50:15.828 INFO:teuthology.orchestra.run.smithi031.stdout: debug buffer = 0/0 2023-11-29T08:50:15.828 INFO:teuthology.orchestra.run.smithi031.stdout: debug civetweb = 0/0 2023-11-29T08:50:15.828 INFO:teuthology.orchestra.run.smithi031.stdout: debug client = 0/0 2023-11-29T08:50:15.828 INFO:teuthology.orchestra.run.smithi031.stdout: debug compressor = 0/0 2023-11-29T08:50:15.828 INFO:teuthology.orchestra.run.smithi031.stdout: debug context = 0/0 2023-11-29T08:50:15.828 INFO:teuthology.orchestra.run.smithi031.stdout: debug crush = 0/0 2023-11-29T08:50:15.828 INFO:teuthology.orchestra.run.smithi031.stdout: debug crypto = 0/0 2023-11-29T08:50:15.828 INFO:teuthology.orchestra.run.smithi031.stdout: debug dpdk = 0/0 2023-11-29T08:50:15.828 INFO:teuthology.orchestra.run.smithi031.stdout: debug eventtrace = 0/0 2023-11-29T08:50:15.828 INFO:teuthology.orchestra.run.smithi031.stdout: debug filer = 0/0 2023-11-29T08:50:15.828 INFO:teuthology.orchestra.run.smithi031.stdout: debug filestore = 0/0 2023-11-29T08:50:15.829 INFO:teuthology.orchestra.run.smithi031.stdout: debug finisher = 0/0 2023-11-29T08:50:15.829 INFO:teuthology.orchestra.run.smithi031.stdout: debug fuse = 0/0 2023-11-29T08:50:15.829 INFO:teuthology.orchestra.run.smithi031.stdout: debug heartbeatmap = 0/0 2023-11-29T08:50:15.829 INFO:teuthology.orchestra.run.smithi031.stdout: debug javaclient = 0/0 2023-11-29T08:50:15.829 INFO:teuthology.orchestra.run.smithi031.stdout: debug journal = 0/0 2023-11-29T08:50:15.829 INFO:teuthology.orchestra.run.smithi031.stdout: debug journaler = 0/0 2023-11-29T08:50:15.829 INFO:teuthology.orchestra.run.smithi031.stdout: debug kstore = 0/0 2023-11-29T08:50:15.829 INFO:teuthology.orchestra.run.smithi031.stdout: debug leveldb = 0/0 2023-11-29T08:50:15.829 INFO:teuthology.orchestra.run.smithi031.stdout: debug lockdep = 0/0 2023-11-29T08:50:15.829 INFO:teuthology.orchestra.run.smithi031.stdout: debug mds = 0/0 2023-11-29T08:50:15.829 INFO:teuthology.orchestra.run.smithi031.stdout: debug mds balancer = 0/0 2023-11-29T08:50:15.829 INFO:teuthology.orchestra.run.smithi031.stdout: debug mds locker = 0/0 2023-11-29T08:50:15.829 INFO:teuthology.orchestra.run.smithi031.stdout: debug mds log = 0/0 2023-11-29T08:50:15.829 INFO:teuthology.orchestra.run.smithi031.stdout: debug mds log expire = 0/0 2023-11-29T08:50:15.829 INFO:teuthology.orchestra.run.smithi031.stdout: debug mds migrator = 0/0 2023-11-29T08:50:15.830 INFO:teuthology.orchestra.run.smithi031.stdout: debug memdb = 0/0 2023-11-29T08:50:15.830 INFO:teuthology.orchestra.run.smithi031.stdout: debug mgr = 0/0 2023-11-29T08:50:15.830 INFO:teuthology.orchestra.run.smithi031.stdout: debug mgrc = 0/0 2023-11-29T08:50:15.830 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 0/0 2023-11-29T08:50:15.830 INFO:teuthology.orchestra.run.smithi031.stdout: debug monc = 0/0 2023-11-29T08:50:15.830 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 0/0 2023-11-29T08:50:15.830 INFO:teuthology.orchestra.run.smithi031.stdout: debug objclass = 0/0 2023-11-29T08:50:15.830 INFO:teuthology.orchestra.run.smithi031.stdout: debug objectcacher = 0/0 2023-11-29T08:50:15.830 INFO:teuthology.orchestra.run.smithi031.stdout: debug objecter = 0/0 2023-11-29T08:50:15.830 INFO:teuthology.orchestra.run.smithi031.stdout: debug optracker = 0/0 2023-11-29T08:50:15.830 INFO:teuthology.orchestra.run.smithi031.stdout: debug osd = 0/0 2023-11-29T08:50:15.830 INFO:teuthology.orchestra.run.smithi031.stdout: debug paxos = 0/0 2023-11-29T08:50:15.830 INFO:teuthology.orchestra.run.smithi031.stdout: debug perfcounter = 0/0 2023-11-29T08:50:15.831 INFO:teuthology.orchestra.run.smithi031.stdout: debug rados = 0/0 2023-11-29T08:50:15.831 INFO:teuthology.orchestra.run.smithi031.stdout: debug rbd = 0/0 2023-11-29T08:50:15.831 INFO:teuthology.orchestra.run.smithi031.stdout: debug rbd mirror = 0/0 2023-11-29T08:50:15.831 INFO:teuthology.orchestra.run.smithi031.stdout: debug rbd replay = 0/0 2023-11-29T08:50:15.831 INFO:teuthology.orchestra.run.smithi031.stdout: debug refs = 0/0 2023-11-29T08:50:15.831 INFO:teuthology.orchestra.run.smithi031.stdout: debug rgw = 0/0 2023-11-29T08:50:15.831 INFO:teuthology.orchestra.run.smithi031.stdout: debug rgw sync = 0/0 2023-11-29T08:50:15.831 INFO:teuthology.orchestra.run.smithi031.stdout: debug rocksdb = 0/0 2023-11-29T08:50:15.831 INFO:teuthology.orchestra.run.smithi031.stdout: debug striper = 0/0 2023-11-29T08:50:15.831 INFO:teuthology.orchestra.run.smithi031.stdout: debug throttle = 0/0 2023-11-29T08:50:15.831 INFO:teuthology.orchestra.run.smithi031.stdout: debug timer = 0/0 2023-11-29T08:50:15.831 INFO:teuthology.orchestra.run.smithi031.stdout: debug tp = 0/0 2023-11-29T08:50:15.831 INFO:teuthology.orchestra.run.smithi031.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-11-29T08:50:15.831 INFO:teuthology.orchestra.run.smithi031.stdout: ms client mode = crc 2023-11-29T08:50:15.831 INFO:teuthology.orchestra.run.smithi031.stdout: ms cluster mode = crc 2023-11-29T08:50:15.832 INFO:teuthology.orchestra.run.smithi031.stdout: ms mon client mode = crc 2023-11-29T08:50:15.832 INFO:teuthology.orchestra.run.smithi031.stdout: ms mon cluster mode = crc 2023-11-29T08:50:15.832 INFO:teuthology.orchestra.run.smithi031.stdout: ms mon service mode = crc 2023-11-29T08:50:15.832 INFO:teuthology.orchestra.run.smithi031.stdout: ms service mode = crc 2023-11-29T08:50:15.832 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = 5ce98019-9fb4-4498-915a-520cb621b53d 2023-11-29T08:50:15.832 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.832 INFO:teuthology.orchestra.run.smithi031.stdout:[osd] 2023-11-29T08:50:15.832 INFO:teuthology.orchestra.run.smithi031.stdout: osd journal size = 100 2023-11-29T08:50:15.832 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.832 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub load threshold = 5.0 2023-11-29T08:50:15.832 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub max interval = 600 2023-11-29T08:50:15.832 INFO:teuthology.orchestra.run.smithi031.stdout: osd mclock profile = high_recovery_ops 2023-11-29T08:50:15.832 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.832 INFO:teuthology.orchestra.run.smithi031.stdout: osd recover clone overlap = true 2023-11-29T08:50:15.832 INFO:teuthology.orchestra.run.smithi031.stdout: osd recovery max chunk = 1048576 2023-11-29T08:50:15.833 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.833 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug shutdown = true 2023-11-29T08:50:15.833 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug op order = true 2023-11-29T08:50:15.833 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug verify stray on activate = true 2023-11-29T08:50:15.833 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug trim objects = true 2023-11-29T08:50:15.833 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.833 INFO:teuthology.orchestra.run.smithi031.stdout: osd open classes on start = true 2023-11-29T08:50:15.833 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug pg log writeout = true 2023-11-29T08:50:15.833 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.833 INFO:teuthology.orchestra.run.smithi031.stdout: osd deep scrub update digest min age = 30 2023-11-29T08:50:15.833 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.833 INFO:teuthology.orchestra.run.smithi031.stdout: osd map max advance = 10 2023-11-29T08:50:15.833 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.833 INFO:teuthology.orchestra.run.smithi031.stdout: journal zero on create = true 2023-11-29T08:50:15.833 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.833 INFO:teuthology.orchestra.run.smithi031.stdout: filestore ondisk finisher threads = 3 2023-11-29T08:50:15.834 INFO:teuthology.orchestra.run.smithi031.stdout: filestore apply finisher threads = 3 2023-11-29T08:50:15.834 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.834 INFO:teuthology.orchestra.run.smithi031.stdout: bdev debug aio = true 2023-11-29T08:50:15.834 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug misdirected ops = true 2023-11-29T08:50:15.834 INFO:teuthology.orchestra.run.smithi031.stdout: debug monc = 20 2023-11-29T08:50:15.834 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore block size = 96636764160 2023-11-29T08:50:15.834 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore compression algorithm = snappy 2023-11-29T08:50:15.834 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore compression mode = aggressive 2023-11-29T08:50:15.834 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore fsck on mount = True 2023-11-29T08:50:15.834 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore rocksdb cf = False 2023-11-29T08:50:15.834 INFO:teuthology.orchestra.run.smithi031.stdout: crimson seastar smp = 3 2023-11-29T08:50:15.834 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluefs = 20 2023-11-29T08:50:15.834 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluestore = 20 2023-11-29T08:50:15.834 INFO:teuthology.orchestra.run.smithi031.stdout: debug filestore = 0/0 2023-11-29T08:50:15.834 INFO:teuthology.orchestra.run.smithi031.stdout: debug journal = 0/0 2023-11-29T08:50:15.835 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 0/0 2023-11-29T08:50:15.835 INFO:teuthology.orchestra.run.smithi031.stdout: debug osd = 0/0 2023-11-29T08:50:15.835 INFO:teuthology.orchestra.run.smithi031.stdout: debug rocksdb = 10 2023-11-29T08:50:15.835 INFO:teuthology.orchestra.run.smithi031.stdout: err to stderr = True 2023-11-29T08:50:15.835 INFO:teuthology.orchestra.run.smithi031.stdout: log flush on exit = True 2023-11-29T08:50:15.835 INFO:teuthology.orchestra.run.smithi031.stdout: log to file = False 2023-11-29T08:50:15.835 INFO:teuthology.orchestra.run.smithi031.stdout: log to stderr = True 2023-11-29T08:50:15.835 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd backfillfull_ratio = 0.85 2023-11-29T08:50:15.835 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd full ratio = 0.9 2023-11-29T08:50:15.835 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd nearfull ratio = 0.8 2023-11-29T08:50:15.835 INFO:teuthology.orchestra.run.smithi031.stdout: osd failsafe full ratio = 0.95 2023-11-29T08:50:15.835 INFO:teuthology.orchestra.run.smithi031.stdout: osd objectstore = bluestore 2023-11-29T08:50:15.835 INFO:teuthology.orchestra.run.smithi031.stdout: osd shutdown pgref assert = True 2023-11-29T08:50:15.835 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.835 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr] 2023-11-29T08:50:15.836 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2023-11-29T08:50:15.836 INFO:teuthology.orchestra.run.smithi031.stdout: debug mgr = 20 2023-11-29T08:50:15.836 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2023-11-29T08:50:15.836 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2023-11-29T08:50:15.836 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2023-11-29T08:50:15.836 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2023-11-29T08:50:15.836 INFO:teuthology.orchestra.run.smithi031.stdout: mgr/telemetry/nag = false 2023-11-29T08:50:15.836 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.836 INFO:teuthology.orchestra.run.smithi031.stdout:[mon] 2023-11-29T08:50:15.836 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 0/0 2023-11-29T08:50:15.836 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 0/0 2023-11-29T08:50:15.836 INFO:teuthology.orchestra.run.smithi031.stdout: debug paxos = 0/0 2023-11-29T08:50:15.836 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2023-11-29T08:50:15.836 INFO:teuthology.orchestra.run.smithi031.stdout: mon data avail warn = 5 2023-11-29T08:50:15.836 INFO:teuthology.orchestra.run.smithi031.stdout: mon mgr mkfs grace = 240 2023-11-29T08:50:15.836 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2023-11-29T08:50:15.837 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd reporter subtree level = osd 2023-11-29T08:50:15.837 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd prime pg temp = true 2023-11-29T08:50:15.837 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2023-11-29T08:50:15.837 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.837 INFO:teuthology.orchestra.run.smithi031.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-29T08:50:15.837 INFO:teuthology.orchestra.run.smithi031.stdout: auth mon ticket ttl = 660 # 11m 2023-11-29T08:50:15.837 INFO:teuthology.orchestra.run.smithi031.stdout: auth service ticket ttl = 240 # 4m 2023-11-29T08:50:15.837 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.837 INFO:teuthology.orchestra.run.smithi031.stdout: # don't complain about insecure global_id in the test suite 2023-11-29T08:50:15.837 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-29T08:50:15.837 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-29T08:50:15.837 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.837 INFO:teuthology.orchestra.run.smithi031.stdout: # 1m isn't quite enough 2023-11-29T08:50:15.837 INFO:teuthology.orchestra.run.smithi031.stdout: mon_down_mkfs_grace = 2m 2023-11-29T08:50:15.837 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.838 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_filestore_osds = false 2023-11-29T08:50:15.838 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default crimson = True 2023-11-29T08:50:15.838 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:15.838 INFO:teuthology.orchestra.run.smithi031.stdout:[client] 2023-11-29T08:50:15.838 INFO:teuthology.orchestra.run.smithi031.stdout: rgw cache enabled = true 2023-11-29T08:50:15.838 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable ops log = true 2023-11-29T08:50:15.838 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable usage log = true 2023-11-29T08:50:15.838 INFO:teuthology.orchestra.run.smithi031.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-29T08:50:15.838 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-29T08:50:15.838 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.a] 2023-11-29T08:50:15.841 INFO:tasks.ceph:Creating admin key on mon.a... 2023-11-29T08:50:15.841 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-11-29T08:50:15.946 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-29T08:50:15.947 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-11-29T08:50:15.947 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-11-29T08:50:15.962 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-11-29T08:50:15.962 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-11-29T08:50:16.017 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:50:16.017 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-11-29T08:50:16.017 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-29T08:50:16.017 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-29T08:50:16.104 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-11-29T08:50:16.104 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-29T08:50:16.160 INFO:tasks.ceph:Setting up mon nodes... 2023-11-29T08:50:16.160 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-29T08:50:16.160 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-11-29T08:50:16.268 INFO:teuthology.orchestra.run.smithi031.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-11-29T08:50:16.271 INFO:tasks.ceph:Setting up mds nodes... 2023-11-29T08:50:16.271 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-29T08:50:16.271 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-29T08:50:16.323 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-11-29T08:50:16.342 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-29T08:50:16.342 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-29T08:50:16.343 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-11-29T08:50:16.373 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-29T08:50:16.373 INFO:tasks.ceph:role: osd.0 2023-11-29T08:50:16.373 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:50:16.373 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-29T08:50:16.451 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-29T08:50:16.451 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-29T08:50:16.451 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-29T08:50:16.451 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-29T08:50:16.452 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-29T08:50:16.452 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2023-11-29T08:50:16.452 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-29T08:50:16.452 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-29T08:50:16.452 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-29T08:50:16.452 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-29T08:50:16.965 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2023-11-29T08:50:16.967 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2023-11-29T08:50:16.968 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-11-29T08:50:17.127 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-11-29T08:50:17.165 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-11-29T08:50:17.238 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-29T08:50:17.238 INFO:tasks.ceph:role: osd.1 2023-11-29T08:50:17.238 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:50:17.238 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-29T08:50:17.317 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-29T08:50:17.317 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-29T08:50:17.317 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-29T08:50:17.317 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-29T08:50:17.317 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-29T08:50:17.317 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2023-11-29T08:50:17.317 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-29T08:50:17.318 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-29T08:50:17.318 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-29T08:50:17.318 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-29T08:50:17.758 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2023-11-29T08:50:17.760 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2023-11-29T08:50:17.761 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-11-29T08:50:17.853 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-11-29T08:50:17.890 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-11-29T08:50:17.961 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-29T08:50:17.961 INFO:tasks.ceph:role: osd.2 2023-11-29T08:50:17.961 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:50:17.961 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-29T08:50:18.039 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-29T08:50:18.039 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-29T08:50:18.039 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-29T08:50:18.039 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-29T08:50:18.039 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-29T08:50:18.039 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2023-11-29T08:50:18.039 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-29T08:50:18.040 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-29T08:50:18.040 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-29T08:50:18.040 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-29T08:50:18.531 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2023-11-29T08:50:18.533 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2023-11-29T08:50:18.533 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-11-29T08:50:18.691 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-11-29T08:50:18.727 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-29T08:50:19.294 INFO:teuthology.orchestra.run.smithi031.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-29T08:50:19.295 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:19,295 seastar - Reactor backend: linux-aio 2023-11-29T08:50:19.312 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:19,311 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-29T08:50:19.312 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:19,311 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-29T08:50:19.312 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:19,311 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-29T08:50:19.313 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:19,311 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-29T08:50:19.314 INFO:teuthology.orchestra.run.smithi031.stderr:==80247==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-29T08:50:19.320 INFO:teuthology.orchestra.run.smithi031.stderr:==80247==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffdc0cae000; bottom 0x7f7bb8329000; size: 0x008208985000 (558489948160) 2023-11-29T08:50:19.321 INFO:teuthology.orchestra.run.smithi031.stderr:False positive error reports may follow 2023-11-29T08:50:19.321 INFO:teuthology.orchestra.run.smithi031.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-29T08:50:19.347 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c8178ec 0x2c818c9a 0x2c81a1e8 0x2c81a56c 0x2c81a6b6 0x2c81ab0c 0x54daf 0x22e5b359 0x22e8816d 0x22e89276 0x22e3d94c 0x22e3dfee 0x22e4dfe7 0x22e50132 0x22e744f4 0x22e61416 0x22e649b0 0x22e64f17 0x22ec8447 0x22ec8cb3 0x22e49f83 0x22e593d0 0x22e5aaaf 0x22ecd494 0x22ece55a 0x22e6b2bc 0x22ed129c 0x22ed31cb 0x22ed4bd0 0x22ed592e 0x22e950d2 0x22d1de8c 0x22b9454a 0x24b532a0 0x24b5aa56 0x24b5d3a4 0x24b5e57d 0x226d8531 0x22716a4a 0x2273cb51 0x2c7bad53 0x2c858b78 0x2ca7ba30 0x2ca7d753 0x2c4d6030 0x2c4d7f39 0x1ee1ef87 0x1ee290f5 0x1ec98990 0x3feaf 0x3ff5f 0x1e743a44 2023-11-29T08:50:19.347 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 2023-11-29T08:50:19.870 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-11-29T08:50:19.870 INFO:teuthology.orchestra.run.smithi031.stderr:Suppressions used: 2023-11-29T08:50:19.870 INFO:teuthology.orchestra.run.smithi031.stderr: count bytes template 2023-11-29T08:50:19.870 INFO:teuthology.orchestra.run.smithi031.stderr: 1 8 MallocExtension::Register 2023-11-29T08:50:19.870 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-11-29T08:50:19.870 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-11-29T08:50:19.928 INFO:teuthology.orchestra.run.smithi031.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-29T08:50:19.928 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:19,928 seastar - Reactor backend: linux-aio 2023-11-29T08:50:19.946 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:19,946 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-29T08:50:19.946 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:19,946 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-29T08:50:19.946 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:19,946 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-29T08:50:19.954 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:19,954 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-29T08:50:19.957 INFO:teuthology.orchestra.run.smithi031.stderr:==80245==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-29T08:50:19.963 INFO:teuthology.orchestra.run.smithi031.stderr:==80245==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffdc0cae000; bottom 0x7f7bb6122000; size: 0x00820ab8c000 (558525628416) 2023-11-29T08:50:19.963 INFO:teuthology.orchestra.run.smithi031.stderr:False positive error reports may follow 2023-11-29T08:50:19.963 INFO:teuthology.orchestra.run.smithi031.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-29T08:50:19.989 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c8178ec 0x2c818c9a 0x2c81a1e8 0x2c81a56c 0x2c81a6b6 0x2c81ab0c 0x54daf 0x1ec5891c 0x1ec59106 0x1ebd34c1 0x22b43ece 0x22b51689 0x22b576d5 0x22b59094 0x24b53d6b 0x24b5aa56 0x24b5d3a4 0x24b5e57d 0x226d8531 0x22716a4a 0x2273cb51 0x2c7bad53 0x2c858b78 0x2ca7ba30 0x2ca7d753 0x2c4d6030 0x2c4d7f39 0x1ec9a246 0x3feaf 0x3ff5f 0x1e743a44 2023-11-29T08:50:19.989 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 2023-11-29T08:50:20.047 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c8178ec 0x2c818c9a 0x2c81a1e8 0x2c81a56c 0x2c81a6b6 0x2c81ab0c 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x263b64c2 0x263b8b0b 0x263ba91c 0x263bd181 0x261e3655 0x1ed52f8c 0x1ecc84bb 0x1eccc2fc 0x2cb9ad8d 2023-11-29T08:50:20.047 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8c5508e5 0xffffffff8d06646c 0xffffffff8d2000ea 2023-11-29T08:50:23.993 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-11-29T08:50:23.994 INFO:teuthology.orchestra.run.smithi031.stderr:Suppressions used: 2023-11-29T08:50:23.994 INFO:teuthology.orchestra.run.smithi031.stderr: count bytes template 2023-11-29T08:50:23.994 INFO:teuthology.orchestra.run.smithi031.stderr: 1 8 MallocExtension::Register 2023-11-29T08:50:23.994 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-11-29T08:50:23.994 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-11-29T08:50:23.994 INFO:teuthology.orchestra.run.smithi031.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 5ce98019-9fb4-4498-915a-520cb621b53d 2023-11-29T08:50:24.031 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-11-29T08:50:24.062 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-29T08:50:24.613 INFO:teuthology.orchestra.run.smithi031.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-29T08:50:24.614 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:24,614 seastar - Reactor backend: linux-aio 2023-11-29T08:50:24.634 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:24,633 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-29T08:50:24.634 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:24,633 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-29T08:50:24.634 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:24,633 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-29T08:50:24.634 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:24,633 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-29T08:50:24.636 INFO:teuthology.orchestra.run.smithi031.stderr:==80429==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-29T08:50:24.642 INFO:teuthology.orchestra.run.smithi031.stderr:==80429==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd54768000; bottom 0x7f0878fd7000; size: 0x00f4db791000 (1051654164480) 2023-11-29T08:50:24.643 INFO:teuthology.orchestra.run.smithi031.stderr:False positive error reports may follow 2023-11-29T08:50:24.643 INFO:teuthology.orchestra.run.smithi031.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-29T08:50:24.668 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c8178ec 0x2c818c9a 0x2c81a1e8 0x2c81a56c 0x2c81a6b6 0x2c81ab0c 0x54daf 0x22e5b137 0x22e8816d 0x22e89276 0x22e3d94c 0x22e3dfee 0x22e3a12b 0x22e3c185 0x22e54aaa 0x22e579e5 0x22e6593a 0x22ec8447 0x22ec8cb3 0x22e49f83 0x22e593d0 0x22e5aaaf 0x22ecd494 0x22ece55a 0x22e6b2bc 0x22ed129c 0x22ed31cb 0x22ed4bd0 0x22ed592e 0x22e950d2 0x22d1de8c 0x22b9454a 0x24b532a0 0x24b5aa56 0x24b5d3a4 0x24b5e57d 0x226d8531 0x22716a4a 0x2273cb51 0x2c7bad53 0x2c858b78 0x2ca7ba30 0x2ca7d753 0x2c4d6030 0x2c4d7f39 0x1ee1ef87 0x1ee290f5 0x1ec98990 0x3feaf 0x3ff5f 0x1e743a44 2023-11-29T08:50:24.668 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 2023-11-29T08:50:25.186 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-11-29T08:50:25.186 INFO:teuthology.orchestra.run.smithi031.stderr:Suppressions used: 2023-11-29T08:50:25.186 INFO:teuthology.orchestra.run.smithi031.stderr: count bytes template 2023-11-29T08:50:25.186 INFO:teuthology.orchestra.run.smithi031.stderr: 1 8 MallocExtension::Register 2023-11-29T08:50:25.186 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-11-29T08:50:25.186 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-11-29T08:50:25.242 INFO:teuthology.orchestra.run.smithi031.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-29T08:50:25.242 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:25,242 seastar - Reactor backend: linux-aio 2023-11-29T08:50:25.260 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:25,260 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-29T08:50:25.260 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:25,260 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-29T08:50:25.261 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:25,260 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-29T08:50:25.268 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:25,267 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-29T08:50:25.270 INFO:teuthology.orchestra.run.smithi031.stderr:==80427==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-29T08:50:25.277 INFO:teuthology.orchestra.run.smithi031.stderr:==80427==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd54768000; bottom 0x7f0876dd0000; size: 0x00f4dd998000 (1051689844736) 2023-11-29T08:50:25.277 INFO:teuthology.orchestra.run.smithi031.stderr:False positive error reports may follow 2023-11-29T08:50:25.277 INFO:teuthology.orchestra.run.smithi031.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-29T08:50:25.302 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c8178ec 0x2c818c9a 0x2c81a1e8 0x2c81a56c 0x2c81a6b6 0x2c81ab0c 0x54daf 0x22afb502 0x22c505e0 0x22b4336f 0x22b51689 0x22b576d5 0x22b59094 0x24b53d6b 0x24b5aa56 0x24b5d3a4 0x24b5e57d 0x226d8531 0x22716a4a 0x2273cb51 0x2c7bad53 0x2c858b78 0x2ca7ba30 0x2ca7d753 0x2c4d6030 0x2c4d7f39 0x1ec9a246 0x3feaf 0x3ff5f 0x1e743a44 2023-11-29T08:50:25.303 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 2023-11-29T08:50:29.326 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-11-29T08:50:29.327 INFO:teuthology.orchestra.run.smithi031.stderr:Suppressions used: 2023-11-29T08:50:29.327 INFO:teuthology.orchestra.run.smithi031.stderr: count bytes template 2023-11-29T08:50:29.327 INFO:teuthology.orchestra.run.smithi031.stderr: 1 8 MallocExtension::Register 2023-11-29T08:50:29.327 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-11-29T08:50:29.327 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-11-29T08:50:29.327 INFO:teuthology.orchestra.run.smithi031.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 5ce98019-9fb4-4498-915a-520cb621b53d 2023-11-29T08:50:29.365 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-11-29T08:50:29.398 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-29T08:50:29.960 INFO:teuthology.orchestra.run.smithi031.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-29T08:50:29.961 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:29,961 seastar - Reactor backend: linux-aio 2023-11-29T08:50:29.977 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:29,976 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-29T08:50:29.977 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:29,976 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-29T08:50:29.977 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:29,976 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-29T08:50:29.977 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:29,976 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-29T08:50:29.979 INFO:teuthology.orchestra.run.smithi031.stderr:==80612==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-29T08:50:29.985 INFO:teuthology.orchestra.run.smithi031.stderr:==80612==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff521e7000; bottom 0x7fa2c5c74000; size: 0x005c8c573000 (397491515392) 2023-11-29T08:50:29.985 INFO:teuthology.orchestra.run.smithi031.stderr:False positive error reports may follow 2023-11-29T08:50:29.985 INFO:teuthology.orchestra.run.smithi031.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-29T08:50:30.013 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c8178ec 0x2c818c9a 0x2c81a1e8 0x2c81a56c 0x2c81a6b6 0x2c81ab0c 0x54daf 0x22e4f6ee 0x22e50132 0x22e744f4 0x22e61416 0x22e649b0 0x22e64f17 0x22ec8447 0x22ec8cb3 0x22e49f83 0x22e593d0 0x22e5aaaf 0x22ecd494 0x22ece55a 0x22e6b2bc 0x22ed129c 0x22ed31cb 0x22ed4bd0 0x22ed592e 0x22e950d2 0x22d1de8c 0x22b9454a 0x24b532a0 0x24b5aa56 0x24b5d3a4 0x24b5e57d 0x226d8531 0x22716a4a 0x2273cb51 0x2c7bad53 0x2c858b78 0x2ca7ba30 0x2ca7d753 0x2c4d6030 0x2c4d7f39 0x1ee1ef87 0x1ee290f5 0x1ec98990 0x3feaf 0x3ff5f 0x1e743a44 2023-11-29T08:50:30.013 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 2023-11-29T08:50:30.517 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-11-29T08:50:30.517 INFO:teuthology.orchestra.run.smithi031.stderr:Suppressions used: 2023-11-29T08:50:30.518 INFO:teuthology.orchestra.run.smithi031.stderr: count bytes template 2023-11-29T08:50:30.518 INFO:teuthology.orchestra.run.smithi031.stderr: 1 8 MallocExtension::Register 2023-11-29T08:50:30.518 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-11-29T08:50:30.518 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-11-29T08:50:30.574 INFO:teuthology.orchestra.run.smithi031.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-29T08:50:30.574 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:30,574 seastar - Reactor backend: linux-aio 2023-11-29T08:50:30.592 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:30,592 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-29T08:50:30.592 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:30,592 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-29T08:50:30.592 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:30,592 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-29T08:50:30.600 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-11-29 08:50:30,600 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-29T08:50:30.602 INFO:teuthology.orchestra.run.smithi031.stderr:==80610==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-29T08:50:30.608 INFO:teuthology.orchestra.run.smithi031.stderr:==80610==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff521e7000; bottom 0x7fa2c3a6d000; size: 0x005c8e77a000 (397527195648) 2023-11-29T08:50:30.608 INFO:teuthology.orchestra.run.smithi031.stderr:False positive error reports may follow 2023-11-29T08:50:30.608 INFO:teuthology.orchestra.run.smithi031.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-29T08:50:30.634 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c8178ec 0x2c818c9a 0x2c81a1e8 0x2c81a56c 0x2c81a6b6 0x2c81ab0c 0x54daf 0x22a387ff 0x22b5938f 0x24b53d6b 0x24b5aa56 0x24b5d3a4 0x24b5e57d 0x226d8531 0x22716a4a 0x2273cb51 0x2c7bad53 0x2c858b78 0x2ca7ba30 0x2ca7d753 0x2c4d6030 0x2c4d7f39 0x1ec9a246 0x3feaf 0x3ff5f 0x1e743a44 2023-11-29T08:50:30.634 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 2023-11-29T08:50:30.695 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c8178ec 0x2c818c9a 0x2c81a1e8 0x2c81a56c 0x2c81a6b6 0x2c81ab0c 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x263b64c2 0x263b8b0b 0x263ba91c 0x263bd181 0x261e3655 0x1ed52f8c 0x1ecc84bb 0x1eccc2fc 0x2cb9ad8d 2023-11-29T08:50:30.695 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8c5508a3 0xffffffff8c5508ea 0xffffffff8d06646c 0xffffffff8d2000ea 2023-11-29T08:50:34.644 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-11-29T08:50:34.645 INFO:teuthology.orchestra.run.smithi031.stderr:Suppressions used: 2023-11-29T08:50:34.645 INFO:teuthology.orchestra.run.smithi031.stderr: count bytes template 2023-11-29T08:50:34.645 INFO:teuthology.orchestra.run.smithi031.stderr: 1 8 MallocExtension::Register 2023-11-29T08:50:34.645 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-11-29T08:50:34.645 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-11-29T08:50:34.645 INFO:teuthology.orchestra.run.smithi031.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 5ce98019-9fb4-4498-915a-520cb621b53d 2023-11-29T08:50:34.683 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-11-29T08:50:34.715 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-29T08:50:34.715 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-11-29T08:50:34.715 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-11-29T08:50:34.786 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-11-29T08:50:34.786 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-11-29T08:50:34.856 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-11-29T08:50:34.857 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-11-29T08:50:34.927 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-11-29T08:50:34.927 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-11-29T08:50:34.997 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-11-29T08:50:34.997 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-11-29T08:50:35.053 INFO:tasks.ceph:Adding keys to all mons... 2023-11-29T08:50:35.053 DEBUG:teuthology.orchestra.run.smithi031:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-29T08:50:35.122 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr.x] 2023-11-29T08:50:35.122 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQBI+2ZlRJ/3DxAA5Fr2S59qdiD1YgW6rf8XTA== 2023-11-29T08:50:35.122 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.0] 2023-11-29T08:50:35.122 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQBM+2Zlw4zAABAA0YmU91qMDxwQlpqdDucC7Q== 2023-11-29T08:50:35.122 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.1] 2023-11-29T08:50:35.123 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQBR+2ZlNqRvExAA7EEAFGkK4ZMBOceY4EpnGA== 2023-11-29T08:50:35.123 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.2] 2023-11-29T08:50:35.123 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQBW+2Zlda8/JxAAfpIDBGNfxTCLIplT57BPMA== 2023-11-29T08:50:35.123 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2023-11-29T08:50:35.123 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQBI+2ZlloBCExAAy3ElWyleo1k6/zNZ2NxtLg== 2023-11-29T08:50:35.124 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-29T08:50:35.226 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-29T08:50:35.288 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-29T08:50:35.349 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-29T08:50:35.436 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-29T08:50:35.501 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-29T08:50:35.501 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-11-29T08:50:35.532 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-29T08:50:35.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:35.638+0000 7f44d0239980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:35.640 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:35.638+0000 7f44d0239980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:35.657 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-11-29T08:50:35.688 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-29T08:50:35.743 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-11-29T08:50:35.744 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-29T08:50:35.744 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-11-29T08:50:35.787 INFO:tasks.ceph.mon.a:Started 2023-11-29T08:50:35.787 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-11-29T08:50:35.787 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-11-29T08:50:35.787 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-11-29T08:50:35.790 INFO:tasks.ceph.mgr.x:Started 2023-11-29T08:50:35.790 INFO:tasks.ceph:Setting crush tunables to default 2023-11-29T08:50:35.790 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd crush tunables default 2023-11-29T08:50:35.874 INFO:tasks.ceph.mon.a.smithi031.stderr:2023-11-29T08:50:35.873+0000 7f369d632980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:35.875 INFO:tasks.ceph.mon.a.smithi031.stderr:2023-11-29T08:50:35.873+0000 7f369d632980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:35.875 INFO:tasks.ceph.mon.a.smithi031.stderr:2023-11-29T08:50:35.873+0000 7f369d632980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:35.875 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:35.874+0000 7f778f23af00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:35.904 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:35.898+0000 7f84a0f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:35.927 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:35.926+0000 7f84a0f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:35.998 INFO:teuthology.orchestra.run.smithi031.stderr:adjusted tunables profile to default 2023-11-29T08:50:36.009 INFO:tasks.ceph:check_enable_crimson: True 2023-11-29T08:50:36.009 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2023-11-29T08:50:36.009 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2023-11-29T08:50:36.079 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:36.077+0000 7f778f23af00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:36.079 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:36.078+0000 7f778f23af00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:36.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:36.112+0000 7fb13406d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:36.120 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:36.118+0000 7fb13406d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:36.200 INFO:tasks.ceph.mon.a.smithi031.stderr:2023-11-29T08:50:36.199+0000 7f36955f1640 -1 WARNING: experimental feature 'crimson' is enabled 2023-11-29T08:50:36.200 INFO:tasks.ceph.mon.a.smithi031.stderr:Please be aware that this feature is experimental, untested, 2023-11-29T08:50:36.200 INFO:tasks.ceph.mon.a.smithi031.stderr:unsupported, and may result in data corruption, data loss, 2023-11-29T08:50:36.200 INFO:tasks.ceph.mon.a.smithi031.stderr:and/or irreparable damage to your cluster. Do not use 2023-11-29T08:50:36.200 INFO:tasks.ceph.mon.a.smithi031.stderr:feature with important data. 2023-11-29T08:50:36.200 INFO:tasks.ceph.mon.a.smithi031.stderr: 2023-11-29T08:50:36.544 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:36.543+0000 7f778f23af00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-29T08:50:36.598 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:36.596+0000 7f778f23af00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-29T08:50:37.013 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-11-29T08:50:37.013 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-11-29T08:50:37.014 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-11-29T08:50:37.044 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-11-29T08:50:37.044 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-11-29T08:50:37.115 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-11-29T08:50:37.115 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-11-29T08:50:37.159 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:37.158+0000 7f778f23af00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-29T08:50:37.186 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd new 24ff2f62-0eb9-4346-8ede-24dd78fb3487 0 2023-11-29T08:50:37.216 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:37.214+0000 7f778f23af00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-29T08:50:37.263 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:37.262+0000 7f778f23af00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-29T08:50:37.321 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:37.320+0000 7f778f23af00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-29T08:50:37.326 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:37.320+0000 7fe90ed1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:37.328 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:37.327+0000 7fe90ed1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:37.366 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:37.365+0000 7f778f23af00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-29T08:50:37.406 INFO:teuthology.orchestra.run.smithi031.stdout:0 2023-11-29T08:50:37.417 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd new 66ec6be7-6bf6-43a7-8b0b-082a274633bb 1 2023-11-29T08:50:37.474 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:37.473+0000 7f778f23af00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-29T08:50:37.527 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:37.521+0000 7fabb106d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:37.529 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:37.528+0000 7fabb106d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:37.552 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:37.551+0000 7f778f23af00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-29T08:50:37.600 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:37.598+0000 7f778f23af00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-29T08:50:37.609 INFO:teuthology.orchestra.run.smithi031.stdout:1 2023-11-29T08:50:37.620 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd new 6863db8a-10b0-47b0-b820-855ba1af0a4b 2 2023-11-29T08:50:37.656 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:37.655+0000 7f778f23af00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-29T08:50:37.707 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:37.706+0000 7f778f23af00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-29T08:50:37.731 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:37.725+0000 7ff38a86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:37.733 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:37.732+0000 7ff38a86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:37.811 INFO:teuthology.orchestra.run.smithi031.stdout:2 2023-11-29T08:50:37.822 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2023-11-29T08:50:37.898 INFO:tasks.ceph.osd.0:Restarting daemon 2023-11-29T08:50:37.898 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2023-11-29T08:50:37.941 INFO:tasks.ceph.osd.0:Started 2023-11-29T08:50:37.941 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2023-11-29T08:50:37.976 INFO:tasks.ceph.osd.1:Restarting daemon 2023-11-29T08:50:37.977 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2023-11-29T08:50:38.020 INFO:tasks.ceph.osd.1:Started 2023-11-29T08:50:38.020 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2023-11-29T08:50:38.059 INFO:tasks.ceph.osd.2:Restarting daemon 2023-11-29T08:50:38.059 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2023-11-29T08:50:38.103 INFO:tasks.ceph.osd.2:Started 2023-11-29T08:50:38.104 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-29T08:50:38.133 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:38.132+0000 7f778f23af00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-29T08:50:38.302 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:38.301+0000 7f778f23af00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-29T08:50:38.317 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:38.308+0000 7f9535b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:38.320 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:38.319+0000 7f9535b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:38.461 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:38.460+0000 7f778f23af00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-29T08:50:38.462 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:38.462 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":6,"fsid":"5ce98019-9fb4-4498-915a-520cb621b53d","created":"2023-11-29T08:50:35.918987+0000","modified":"2023-11-29T08:50:37.810233+0000","last_up_change":"0.000000","last_in_change":"2023-11-29T08:50:37.810233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"24ff2f62-0eb9-4346-8ede-24dd78fb3487","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"66ec6be7-6bf6-43a7-8b0b-082a274633bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6863db8a-10b0-47b0-b820-855ba1af0a4b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-29T08:50:38.485 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-11-29T08:50:38.485 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-29T08:50:38.518 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:38.517+0000 7f778f23af00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-29T08:50:38.649 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:38.647+0000 7f778f23af00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-29T08:50:38.751 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:38.750+0000 7f778f23af00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-29T08:50:38.787 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-29T08:50:38.890 INFO:teuthology.misc.health.smithi031.stderr:2023-11-29T08:50:38.885+0000 7fc5266ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:38.892 INFO:teuthology.misc.health.smithi031.stderr:2023-11-29T08:50:38.890+0000 7fc5266ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:38.908 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:38.907+0000 7f778f23af00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-29T08:50:38.956 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:38.954+0000 7f778f23af00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-29T08:50:38.959 INFO:teuthology.misc.health.smithi031.stdout: 2023-11-29T08:50:38.959 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":6,"fsid":"5ce98019-9fb4-4498-915a-520cb621b53d","created":"2023-11-29T08:50:35.918987+0000","modified":"2023-11-29T08:50:37.810233+0000","last_up_change":"0.000000","last_in_change":"2023-11-29T08:50:37.810233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"24ff2f62-0eb9-4346-8ede-24dd78fb3487","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"66ec6be7-6bf6-43a7-8b0b-082a274633bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6863db8a-10b0-47b0-b820-855ba1af0a4b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-29T08:50:38.969 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-11-29T08:50:39.113 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:39.112+0000 7f778f23af00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-29T08:50:39.970 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:39.969+0000 7f778f23af00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-29T08:50:40.283 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:40.282+0000 7f778f23af00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-29T08:50:40.343 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:40.341+0000 7f778f23af00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-29T08:50:40.453 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:40.451+0000 7f778f23af00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-29T08:50:41.473 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-11-29T08:50:41.471+0000 7f772f33d640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-11-29T08:50:45.272 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-29T08:50:45.383 INFO:teuthology.misc.health.smithi031.stderr:2023-11-29T08:50:45.378+0000 7f2677a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:45.385 INFO:teuthology.misc.health.smithi031.stderr:2023-11-29T08:50:45.384+0000 7f2677a6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:45.593 INFO:teuthology.misc.health.smithi031.stdout: 2023-11-29T08:50:45.593 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":9,"fsid":"5ce98019-9fb4-4498-915a-520cb621b53d","created":"2023-11-29T08:50:35.918987+0000","modified":"2023-11-29T08:50:45.479516+0000","last_up_change":"2023-11-29T08:50:43.473526+0000","last_in_change":"2023-11-29T08:50:37.810233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-29T08:50:44.524254+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24ff2f62-0eb9-4346-8ede-24dd78fb3487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":4138680940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":4138680940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6813","nonce":4138680940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":4138680940}]},"public_addr":"172.21.15.31:6806/4138680940","cluster_addr":"172.21.15.31:6807/4138680940","heartbeat_back_addr":"172.21.15.31:6813/4138680940","heartbeat_front_addr":"172.21.15.31:6812/4138680940","state":["exists","up"]},{"osd":1,"uuid":"66ec6be7-6bf6-43a7-8b0b-082a274633bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":2698659858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2698659858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2698659858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":2698659858}]},"public_addr":"172.21.15.31:6803/2698659858","cluster_addr":"172.21.15.31:6802/2698659858","heartbeat_back_addr":"172.21.15.31:6808/2698659858","heartbeat_front_addr":"172.21.15.31:6811/2698659858","state":["exists","up"]},{"osd":2,"uuid":"6863db8a-10b0-47b0-b820-855ba1af0a4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":99605352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":99605352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":99605352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":99605352}]},"public_addr":"172.21.15.31:6804/99605352","cluster_addr":"172.21.15.31:6805/99605352","heartbeat_back_addr":"172.21.15.31:6810/99605352","heartbeat_front_addr":"172.21.15.31:6809/99605352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-29T08:50:45.603 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-11-29T08:50:45.604 INFO:tasks.ceph:Creating RBD pool 2023-11-29T08:50:45.604 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd pool create rbd 8 2023-11-29T08:50:45.715 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:45.708+0000 7f4576cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:45.717 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:45.716+0000 7f4576cad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:46.486 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'rbd' created 2023-11-29T08:50:46.508 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-11-29T08:50:46.639 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:46.629+0000 7fcb9f35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:46.641 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:46.640+0000 7fcb9f35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:47.496 INFO:teuthology.orchestra.run.smithi031.stderr:enabled application 'rbd' on pool 'rbd' 2023-11-29T08:50:47.510 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-11-29T08:50:47.510 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-29T08:50:47.510 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-11-29T08:50:47.511 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-11-29T08:50:47.511 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-29T08:50:47.648 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:47.643+0000 7ffb35f15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:47.650 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:47.649+0000 7ffb35f15640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:47.887 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:47.897 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3173753327},{"type":"v1","addr":"172.21.15.31:6801","nonce":3173753327}]},"active_addr":"172.21.15.31:6801/3173753327","active_change":"2023-11-29T08:50:40.465853+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":3335633426}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":672218365}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":517032724}]}]} 2023-11-29T08:50:47.901 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-11-29T08:50:47.901 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-11-29T08:50:47.901 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-29T08:50:48.013 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:48.009+0000 7f0a27f11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:48.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:48.014+0000 7f0a27f11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:48.202 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:48.202 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":11,"fsid":"5ce98019-9fb4-4498-915a-520cb621b53d","created":"2023-11-29T08:50:35.918987+0000","modified":"2023-11-29T08:50:47.491704+0000","last_up_change":"2023-11-29T08:50:43.473526+0000","last_in_change":"2023-11-29T08:50:37.810233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-29T08:50:44.524254+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-29T08:50:45.928728+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24ff2f62-0eb9-4346-8ede-24dd78fb3487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":4138680940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":4138680940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6813","nonce":4138680940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":4138680940}]},"public_addr":"172.21.15.31:6806/4138680940","cluster_addr":"172.21.15.31:6807/4138680940","heartbeat_back_addr":"172.21.15.31:6813/4138680940","heartbeat_front_addr":"172.21.15.31:6812/4138680940","state":["exists","up"]},{"osd":1,"uuid":"66ec6be7-6bf6-43a7-8b0b-082a274633bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":2698659858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2698659858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2698659858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":2698659858}]},"public_addr":"172.21.15.31:6803/2698659858","cluster_addr":"172.21.15.31:6802/2698659858","heartbeat_back_addr":"172.21.15.31:6808/2698659858","heartbeat_front_addr":"172.21.15.31:6811/2698659858","state":["exists","up"]},{"osd":2,"uuid":"6863db8a-10b0-47b0-b820-855ba1af0a4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":99605352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":99605352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":99605352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":99605352}]},"public_addr":"172.21.15.31:6804/99605352","cluster_addr":"172.21.15.31:6805/99605352","heartbeat_back_addr":"172.21.15.31:6810/99605352","heartbeat_front_addr":"172.21.15.31:6809/99605352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-29T08:50:48.214 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-11-29T08:50:48.215 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-29T08:50:48.327 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:48.322+0000 7faafbead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:48.329 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:48.327+0000 7faafbead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:48.544 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:48.544 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":12,"fsid":"5ce98019-9fb4-4498-915a-520cb621b53d","created":"2023-11-29T08:50:35.918987+0000","modified":"2023-11-29T08:50:48.495373+0000","last_up_change":"2023-11-29T08:50:43.473526+0000","last_in_change":"2023-11-29T08:50:37.810233+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-29T08:50:44.524254+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-29T08:50:45.928728+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"24ff2f62-0eb9-4346-8ede-24dd78fb3487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":4138680940}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":4138680940}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6813","nonce":4138680940}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":4138680940}]},"public_addr":"172.21.15.31:6806/4138680940","cluster_addr":"172.21.15.31:6807/4138680940","heartbeat_back_addr":"172.21.15.31:6813/4138680940","heartbeat_front_addr":"172.21.15.31:6812/4138680940","state":["exists","up"]},{"osd":1,"uuid":"66ec6be7-6bf6-43a7-8b0b-082a274633bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":2698659858}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2698659858}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2698659858}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":2698659858}]},"public_addr":"172.21.15.31:6803/2698659858","cluster_addr":"172.21.15.31:6802/2698659858","heartbeat_back_addr":"172.21.15.31:6808/2698659858","heartbeat_front_addr":"172.21.15.31:6811/2698659858","state":["exists","up"]},{"osd":2,"uuid":"6863db8a-10b0-47b0-b820-855ba1af0a4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":99605352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":99605352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":99605352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":99605352}]},"public_addr":"172.21.15.31:6804/99605352","cluster_addr":"172.21.15.31:6805/99605352","heartbeat_back_addr":"172.21.15.31:6810/99605352","heartbeat_front_addr":"172.21.15.31:6809/99605352","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-29T08:50:48.558 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-11-29T08:50:48.558 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-11-29T08:50:48.559 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-11-29T08:50:48.676 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:48.671+0000 7f23ead9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:48.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:48.671+0000 7f3ec859e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:48.677 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:48.671+0000 7f48bd46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:48.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:48.678+0000 7f3ec859e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:48.679 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:48.678+0000 7f48bd46d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:48.680 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:48.678+0000 7f23ead9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:48.702 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-11-29T08:50:48.703 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-29T08:50:48.707 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-11-29T08:50:48.707 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-29T08:50:48.709 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-11-29T08:50:48.710 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-29T08:50:48.819 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:48.814+0000 7fba26c06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:48.821 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:48.820+0000 7fba26c06640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:48.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:48.860+0000 7f3cf282d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:48.866 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:48.860+0000 7f02b855e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:48.868 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:48.867+0000 7f3cf282d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:48.869 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:48.867+0000 7f02b855e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:49.015 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-11-29T08:50:49.028 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2023-11-29T08:50:49.028 DEBUG:teuthology.parallel:result is None 2023-11-29T08:50:49.055 INFO:teuthology.orchestra.run.smithi031.stdout:34359738372 2023-11-29T08:50:49.058 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-11-29T08:50:49.068 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.0 2023-11-29T08:50:49.071 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2023-11-29T08:50:49.071 DEBUG:teuthology.parallel:result is None 2023-11-29T08:50:50.069 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-29T08:50:50.195 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:50.190+0000 7eff0782d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:50.198 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:50.196+0000 7eff0782d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:50.381 INFO:teuthology.orchestra.run.smithi031.stdout:34359738372 2023-11-29T08:50:50.393 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.0 2023-11-29T08:50:51.394 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-29T08:50:51.521 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:51.516+0000 7fe27a42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:51.523 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:51.522+0000 7fe27a42d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:51.710 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-11-29T08:50:51.723 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2023-11-29T08:50:51.723 DEBUG:teuthology.parallel:result is None 2023-11-29T08:50:51.723 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-29T08:50:51.724 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-29T08:50:51.836 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:51.831+0000 7f5821d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:51.838 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:51.836+0000 7f5821d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:52.018 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:52.018 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-11-29T08:50:52.029 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-11-29T08:50:50.470156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80760,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034760,"statfs":{"total":289910292480,"available":289827594240,"internally_reserved":0,"allocated":270336,"data_stored":1224791,"data_compressed":59930,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.930634"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-29T08:50:48.537104+0000","last_change":"2023-11-29T08:50:47.621004+0000","last_active":"2023-11-29T08:50:48.537104+0000","last_peered":"2023-11-29T08:50:48.537104+0000","last_clean":"2023-11-29T08:50:48.537104+0000","last_became_active":"2023-11-29T08:50:47.616980+0000","last_became_peered":"2023-11-29T08:50:47.616980+0000","last_unstale":"2023-11-29T08:50:48.537104+0000","last_undegraded":"2023-11-29T08:50:48.537104+0000","last_fullsized":"2023-11-29T08:50:48.537104+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_clean_scrub_stamp":"2023-11-29T08:50:46.485331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-29T08:50:48.538085+0000","last_change":"2023-11-29T08:50:47.637168+0000","last_active":"2023-11-29T08:50:48.538085+0000","last_peered":"2023-11-29T08:50:48.538085+0000","last_clean":"2023-11-29T08:50:48.538085+0000","last_became_active":"2023-11-29T08:50:47.634587+0000","last_became_peered":"2023-11-29T08:50:47.634587+0000","last_unstale":"2023-11-29T08:50:48.538085+0000","last_undegraded":"2023-11-29T08:50:48.538085+0000","last_fullsized":"2023-11-29T08:50:48.538085+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_clean_scrub_stamp":"2023-11-29T08:50:46.485331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-29T08:50:48.518953+0000","last_change":"2023-11-29T08:50:47.613890+0000","last_active":"2023-11-29T08:50:48.518953+0000","last_peered":"2023-11-29T08:50:48.518953+0000","last_clean":"2023-11-29T08:50:48.518953+0000","last_became_active":"2023-11-29T08:50:47.611653+0000","last_became_peered":"2023-11-29T08:50:47.611653+0000","last_unstale":"2023-11-29T08:50:48.518953+0000","last_undegraded":"2023-11-29T08:50:48.518953+0000","last_fullsized":"2023-11-29T08:50:48.518953+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_clean_scrub_stamp":"2023-11-29T08:50:46.485331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-29T08:50:48.538555+0000","last_change":"2023-11-29T08:50:47.625921+0000","last_active":"2023-11-29T08:50:48.538555+0000","last_peered":"2023-11-29T08:50:48.538555+0000","last_clean":"2023-11-29T08:50:48.538555+0000","last_became_active":"2023-11-29T08:50:47.612373+0000","last_became_peered":"2023-11-29T08:50:47.612373+0000","last_unstale":"2023-11-29T08:50:48.538555+0000","last_undegraded":"2023-11-29T08:50:48.538555+0000","last_fullsized":"2023-11-29T08:50:48.538555+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_clean_scrub_stamp":"2023-11-29T08:50:46.485331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-29T08:50:48.542206+0000","last_change":"2023-11-29T08:50:47.587943+0000","last_active":"2023-11-29T08:50:48.542206+0000","last_peered":"2023-11-29T08:50:48.542206+0000","last_clean":"2023-11-29T08:50:48.542206+0000","last_became_active":"2023-11-29T08:50:47.584210+0000","last_became_peered":"2023-11-29T08:50:47.584210+0000","last_unstale":"2023-11-29T08:50:48.542206+0000","last_undegraded":"2023-11-29T08:50:48.542206+0000","last_fullsized":"2023-11-29T08:50:48.542206+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_clean_scrub_stamp":"2023-11-29T08:50:46.485331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-29T08:50:48.506350+0000","last_change":"2023-11-29T08:50:47.594228+0000","last_active":"2023-11-29T08:50:48.506350+0000","last_peered":"2023-11-29T08:50:48.506350+0000","last_clean":"2023-11-29T08:50:48.506350+0000","last_became_active":"2023-11-29T08:50:47.591881+0000","last_became_peered":"2023-11-29T08:50:47.591881+0000","last_unstale":"2023-11-29T08:50:48.506350+0000","last_undegraded":"2023-11-29T08:50:48.506350+0000","last_fullsized":"2023-11-29T08:50:48.506350+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_clean_scrub_stamp":"2023-11-29T08:50:46.485331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-29T08:50:48.506143+0000","last_change":"2023-11-29T08:50:47.600260+0000","last_active":"2023-11-29T08:50:48.506143+0000","last_peered":"2023-11-29T08:50:48.506143+0000","last_clean":"2023-11-29T08:50:48.506143+0000","last_became_active":"2023-11-29T08:50:47.598213+0000","last_became_peered":"2023-11-29T08:50:47.598213+0000","last_unstale":"2023-11-29T08:50:48.506143+0000","last_undegraded":"2023-11-29T08:50:48.506143+0000","last_fullsized":"2023-11-29T08:50:48.506143+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_clean_scrub_stamp":"2023-11-29T08:50:46.485331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-29T08:50:48.517601+0000","last_change":"2023-11-29T08:50:47.586184+0000","last_active":"2023-11-29T08:50:48.517601+0000","last_peered":"2023-11-29T08:50:48.517601+0000","last_clean":"2023-11-29T08:50:48.517601+0000","last_became_active":"2023-11-29T08:50:47.582188+0000","last_became_peered":"2023-11-29T08:50:47.582188+0000","last_unstale":"2023-11-29T08:50:48.517601+0000","last_undegraded":"2023-11-29T08:50:48.517601+0000","last_fullsized":"2023-11-29T08:50:48.517601+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_clean_scrub_stamp":"2023-11-29T08:50:46.485331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-29T08:50:48.536639+0000","last_change":"2023-11-29T08:50:46.557331+0000","last_active":"2023-11-29T08:50:48.536639+0000","last_peered":"2023-11-29T08:50:48.536639+0000","last_clean":"2023-11-29T08:50:48.536639+0000","last_became_active":"2023-11-29T08:50:46.545882+0000","last_became_peered":"2023-11-29T08:50:46.545882+0000","last_unstale":"2023-11-29T08:50:48.536639+0000","last_undegraded":"2023-11-29T08:50:48.536639+0000","last_fullsized":"2023-11-29T08:50:48.536639+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-29T08:50:45.479516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-29T08:50:45.479516+0000","last_clean_scrub_stamp":"2023-11-29T08:50:45.479516+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26872,"kb_used_data":48,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344968,"statfs":{"total":96636764160,"available":96609247232,"internally_reserved":0,"allocated":49152,"data_stored":14685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":29965,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":29965,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-29T08:50:52.030 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-29T08:50:52.142 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:52.138+0000 7f37dc35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:52.144 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:52.143+0000 7f37dc35e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:52.324 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:52.324 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-11-29T08:50:52.336 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-11-29T08:50:50.470156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80760,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034760,"statfs":{"total":289910292480,"available":289827594240,"internally_reserved":0,"allocated":270336,"data_stored":1224791,"data_compressed":59930,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.930634"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-29T08:50:48.537104+0000","last_change":"2023-11-29T08:50:47.621004+0000","last_active":"2023-11-29T08:50:48.537104+0000","last_peered":"2023-11-29T08:50:48.537104+0000","last_clean":"2023-11-29T08:50:48.537104+0000","last_became_active":"2023-11-29T08:50:47.616980+0000","last_became_peered":"2023-11-29T08:50:47.616980+0000","last_unstale":"2023-11-29T08:50:48.537104+0000","last_undegraded":"2023-11-29T08:50:48.537104+0000","last_fullsized":"2023-11-29T08:50:48.537104+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_clean_scrub_stamp":"2023-11-29T08:50:46.485331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-29T08:50:48.538085+0000","last_change":"2023-11-29T08:50:47.637168+0000","last_active":"2023-11-29T08:50:48.538085+0000","last_peered":"2023-11-29T08:50:48.538085+0000","last_clean":"2023-11-29T08:50:48.538085+0000","last_became_active":"2023-11-29T08:50:47.634587+0000","last_became_peered":"2023-11-29T08:50:47.634587+0000","last_unstale":"2023-11-29T08:50:48.538085+0000","last_undegraded":"2023-11-29T08:50:48.538085+0000","last_fullsized":"2023-11-29T08:50:48.538085+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_clean_scrub_stamp":"2023-11-29T08:50:46.485331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-29T08:50:48.518953+0000","last_change":"2023-11-29T08:50:47.613890+0000","last_active":"2023-11-29T08:50:48.518953+0000","last_peered":"2023-11-29T08:50:48.518953+0000","last_clean":"2023-11-29T08:50:48.518953+0000","last_became_active":"2023-11-29T08:50:47.611653+0000","last_became_peered":"2023-11-29T08:50:47.611653+0000","last_unstale":"2023-11-29T08:50:48.518953+0000","last_undegraded":"2023-11-29T08:50:48.518953+0000","last_fullsized":"2023-11-29T08:50:48.518953+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_clean_scrub_stamp":"2023-11-29T08:50:46.485331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-29T08:50:48.538555+0000","last_change":"2023-11-29T08:50:47.625921+0000","last_active":"2023-11-29T08:50:48.538555+0000","last_peered":"2023-11-29T08:50:48.538555+0000","last_clean":"2023-11-29T08:50:48.538555+0000","last_became_active":"2023-11-29T08:50:47.612373+0000","last_became_peered":"2023-11-29T08:50:47.612373+0000","last_unstale":"2023-11-29T08:50:48.538555+0000","last_undegraded":"2023-11-29T08:50:48.538555+0000","last_fullsized":"2023-11-29T08:50:48.538555+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_clean_scrub_stamp":"2023-11-29T08:50:46.485331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-29T08:50:48.542206+0000","last_change":"2023-11-29T08:50:47.587943+0000","last_active":"2023-11-29T08:50:48.542206+0000","last_peered":"2023-11-29T08:50:48.542206+0000","last_clean":"2023-11-29T08:50:48.542206+0000","last_became_active":"2023-11-29T08:50:47.584210+0000","last_became_peered":"2023-11-29T08:50:47.584210+0000","last_unstale":"2023-11-29T08:50:48.542206+0000","last_undegraded":"2023-11-29T08:50:48.542206+0000","last_fullsized":"2023-11-29T08:50:48.542206+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_clean_scrub_stamp":"2023-11-29T08:50:46.485331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-29T08:50:48.506350+0000","last_change":"2023-11-29T08:50:47.594228+0000","last_active":"2023-11-29T08:50:48.506350+0000","last_peered":"2023-11-29T08:50:48.506350+0000","last_clean":"2023-11-29T08:50:48.506350+0000","last_became_active":"2023-11-29T08:50:47.591881+0000","last_became_peered":"2023-11-29T08:50:47.591881+0000","last_unstale":"2023-11-29T08:50:48.506350+0000","last_undegraded":"2023-11-29T08:50:48.506350+0000","last_fullsized":"2023-11-29T08:50:48.506350+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_clean_scrub_stamp":"2023-11-29T08:50:46.485331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-29T08:50:48.506143+0000","last_change":"2023-11-29T08:50:47.600260+0000","last_active":"2023-11-29T08:50:48.506143+0000","last_peered":"2023-11-29T08:50:48.506143+0000","last_clean":"2023-11-29T08:50:48.506143+0000","last_became_active":"2023-11-29T08:50:47.598213+0000","last_became_peered":"2023-11-29T08:50:47.598213+0000","last_unstale":"2023-11-29T08:50:48.506143+0000","last_undegraded":"2023-11-29T08:50:48.506143+0000","last_fullsized":"2023-11-29T08:50:48.506143+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_clean_scrub_stamp":"2023-11-29T08:50:46.485331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-29T08:50:48.517601+0000","last_change":"2023-11-29T08:50:47.586184+0000","last_active":"2023-11-29T08:50:48.517601+0000","last_peered":"2023-11-29T08:50:48.517601+0000","last_clean":"2023-11-29T08:50:48.517601+0000","last_became_active":"2023-11-29T08:50:47.582188+0000","last_became_peered":"2023-11-29T08:50:47.582188+0000","last_unstale":"2023-11-29T08:50:48.517601+0000","last_undegraded":"2023-11-29T08:50:48.517601+0000","last_fullsized":"2023-11-29T08:50:48.517601+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-29T08:50:46.485331+0000","last_clean_scrub_stamp":"2023-11-29T08:50:46.485331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-29T08:50:48.536639+0000","last_change":"2023-11-29T08:50:46.557331+0000","last_active":"2023-11-29T08:50:48.536639+0000","last_peered":"2023-11-29T08:50:48.536639+0000","last_clean":"2023-11-29T08:50:48.536639+0000","last_became_active":"2023-11-29T08:50:46.545882+0000","last_became_peered":"2023-11-29T08:50:46.545882+0000","last_unstale":"2023-11-29T08:50:48.536639+0000","last_undegraded":"2023-11-29T08:50:48.536639+0000","last_fullsized":"2023-11-29T08:50:48.536639+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-29T08:50:45.479516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-29T08:50:45.479516+0000","last_clean_scrub_stamp":"2023-11-29T08:50:45.479516+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26872,"kb_used_data":48,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344968,"statfs":{"total":96636764160,"available":96609247232,"internally_reserved":0,"allocated":49152,"data_stored":14685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":29965,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":29965,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-29T08:50:52.336 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-29T08:50:52.337 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-29T08:50:52.337 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-11-29T08:50:52.337 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-29T08:50:52.447 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:52.444+0000 7f7a2e06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:52.449 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:50:52.448+0000 7f7a2e06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:50:52.667 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:52.667 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-29T08:50:52.679 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-11-29T08:50:52.679 INFO:teuthology.run_tasks:Running task cbt... 2023-11-29T08:50:52.692 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi031.front.sepia.ceph.com', 'osds': ['ubuntu@smithi031.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi031.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi031.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi031.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randwrite'], 'norandommap': True, 'op_size': [4194304], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2], 'cmd_path': '/home/ubuntu/cephtest/fio/fio'}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2023-11-29T08:50:52.692 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2023-11-29T08:50:52.710 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-11-29T08:50:52.710 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-11-29T08:50:52.762 DEBUG:teuthology.orchestra.run.smithi031:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2023-11-29T08:50:52.817 INFO:teuthology.orchestra.run.smithi031.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2023-11-29T08:50:53.242 INFO:tasks.cbt:Installing collectl 2023-11-29T08:50:53.242 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2023-11-29T08:50:53.355 INFO:teuthology.orchestra.run.smithi031.stderr:--2023-11-29 08:50:53-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2023-11-29T08:50:53.513 INFO:teuthology.orchestra.run.smithi031.stderr:Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::6812:256f, ... 2023-11-29T08:50:53.522 INFO:teuthology.orchestra.run.smithi031.stderr:Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. 2023-11-29T08:50:53.736 INFO:teuthology.orchestra.run.smithi031.stderr:HTTP request sent, awaiting response... 302 Found 2023-11-29T08:50:53.736 INFO:teuthology.orchestra.run.smithi031.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlZvttB3D2J8y08UTZQeBRJ-6SipiXBRiEcNIgex5W5xtb9gKyft9n9-fID1AUXNNofdM5cVtu0ysct1fO6Ah90B4x2w%3D%3D&use_mirror=netactuate&r= [following] 2023-11-29T08:50:53.736 INFO:teuthology.orchestra.run.smithi031.stderr:--2023-11-29 08:50:53-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlZvttB3D2J8y08UTZQeBRJ-6SipiXBRiEcNIgex5W5xtb9gKyft9n9-fID1AUXNNofdM5cVtu0ysct1fO6Ah90B4x2w%3D%3D&use_mirror=netactuate&r= 2023-11-29T08:50:53.747 INFO:teuthology.orchestra.run.smithi031.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2023-11-29T08:50:53.818 INFO:teuthology.orchestra.run.smithi031.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2023-11-29T08:50:54.052 INFO:teuthology.orchestra.run.smithi031.stderr:HTTP request sent, awaiting response... 302 Found 2023-11-29T08:50:54.052 INFO:teuthology.orchestra.run.smithi031.stderr:Location: https://netactuate.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2023-11-29T08:50:54.052 INFO:teuthology.orchestra.run.smithi031.stderr:--2023-11-29 08:50:54-- https://netactuate.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2023-11-29T08:50:54.064 INFO:teuthology.orchestra.run.smithi031.stderr:Resolving netactuate.dl.sourceforge.net (netactuate.dl.sourceforge.net)... 104.225.3.66 2023-11-29T08:50:54.078 INFO:teuthology.orchestra.run.smithi031.stderr:Connecting to netactuate.dl.sourceforge.net (netactuate.dl.sourceforge.net)|104.225.3.66|:443... connected. 2023-11-29T08:50:54.194 INFO:teuthology.orchestra.run.smithi031.stderr:HTTP request sent, awaiting response... 200 OK 2023-11-29T08:50:54.194 INFO:teuthology.orchestra.run.smithi031.stderr:Length: 620693 (606K) [application/x-gzip] 2023-11-29T08:50:54.194 INFO:teuthology.orchestra.run.smithi031.stderr:Saving to: ‘collectl.tar.gz’ 2023-11-29T08:50:54.194 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-11-29T08:50:54.206 INFO:teuthology.orchestra.run.smithi031.stderr: 0K .......... .......... .......... .......... .......... 8% 3.88M 0s 2023-11-29T08:50:54.219 INFO:teuthology.orchestra.run.smithi031.stderr: 50K .......... .......... .......... .......... .......... 16% 3.90M 0s 2023-11-29T08:50:54.219 INFO:teuthology.orchestra.run.smithi031.stderr: 100K .......... .......... .......... .......... .......... 24% 249M 0s 2023-11-29T08:50:54.233 INFO:teuthology.orchestra.run.smithi031.stderr: 150K .......... .......... .......... .......... .......... 32% 3.87M 0s 2023-11-29T08:50:54.233 INFO:teuthology.orchestra.run.smithi031.stderr: 200K .......... .......... .......... .......... .......... 41% 327M 0s 2023-11-29T08:50:54.235 INFO:teuthology.orchestra.run.smithi031.stderr: 250K .......... .......... .......... .......... .......... 49% 14.4M 0s 2023-11-29T08:50:54.246 INFO:teuthology.orchestra.run.smithi031.stderr: 300K .......... .......... .......... .......... .......... 57% 5.44M 0s 2023-11-29T08:50:54.246 INFO:teuthology.orchestra.run.smithi031.stderr: 350K .......... .......... .......... .......... .......... 65% 251M 0s 2023-11-29T08:50:54.246 INFO:teuthology.orchestra.run.smithi031.stderr: 400K .......... .......... .......... .......... .......... 74% 321M 0s 2023-11-29T08:50:54.246 INFO:teuthology.orchestra.run.smithi031.stderr: 450K .......... .......... .......... .......... .......... 82% 322M 0s 2023-11-29T08:50:54.248 INFO:teuthology.orchestra.run.smithi031.stderr: 500K .......... .......... .......... .......... .......... 90% 15.5M 0s 2023-11-29T08:50:54.248 INFO:teuthology.orchestra.run.smithi031.stderr: 550K .......... .......... .......... .......... .......... 98% 251M 0s 2023-11-29T08:50:54.248 INFO:teuthology.orchestra.run.smithi031.stderr: 600K ...... 100% 11.4T=0.05s 2023-11-29T08:50:54.249 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-11-29T08:50:54.249 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29 08:50:54 (10.9 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2023-11-29T08:50:54.249 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-11-29T08:50:54.311 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/ 2023-11-29T08:50:54.312 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/collectl.conf 2023-11-29T08:50:54.312 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/ 2023-11-29T08:50:54.312 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/SlowProc.html 2023-11-29T08:50:54.312 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/BuddyInfo.html 2023-11-29T08:50:54.312 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/WhySummary.html 2023-11-29T08:50:54.312 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Socket.html 2023-11-29T08:50:54.312 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/colmux.html 2023-11-29T08:50:54.312 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Tutorial.html 2023-11-29T08:50:54.312 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Gexpr.html 2023-11-29T08:50:54.312 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Import.html 2023-11-29T08:50:54.312 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/style.css 2023-11-29T08:50:54.312 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2023-11-29T08:50:54.312 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2023-11-29T08:50:54.312 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/FileNaming.html 2023-11-29T08:50:54.313 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/TheMath.html 2023-11-29T08:50:54.313 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Environmental.html 2023-11-29T08:50:54.313 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Process.html 2023-11-29T08:50:54.313 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/OutputFiles.html 2023-11-29T08:50:54.313 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/HiResTime.html 2023-11-29T08:50:54.313 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/OutputFormats.html 2023-11-29T08:50:54.313 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/slub.jpg 2023-11-29T08:50:54.313 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Lustre.html 2023-11-29T08:50:54.313 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Documentation.html 2023-11-29T08:50:54.313 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Inodes.html 2023-11-29T08:50:54.313 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Graphite.html 2023-11-29T08:50:54.313 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/InputFiles.html 2023-11-29T08:50:54.313 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Export.html 2023-11-29T08:50:54.313 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Interrupts.html 2023-11-29T08:50:54.313 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Startup.html 2023-11-29T08:50:54.313 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Performance.html 2023-11-29T08:50:54.314 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Messages.html 2023-11-29T08:50:54.314 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Logging.html 2023-11-29T08:50:54.314 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Data-detail.html 2023-11-29T08:50:54.314 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/CPUs.html 2023-11-29T08:50:54.314 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Matrix.html 2023-11-29T08:50:54.314 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Data-brief.html 2023-11-29T08:50:54.314 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/SlabInfo.html 2023-11-29T08:50:54.314 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Misc.html 2023-11-29T08:50:54.314 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Examples.html 2023-11-29T08:50:54.314 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/OpenStack.html 2023-11-29T08:50:54.314 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Plotfiles.html 2023-11-29T08:50:54.314 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2023-11-29T08:50:54.314 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2023-11-29T08:50:54.314 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Data-verbose.html 2023-11-29T08:50:54.314 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Data.html 2023-11-29T08:50:54.314 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Exceptions.html 2023-11-29T08:50:54.315 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/RunningAsAService.html 2023-11-29T08:50:54.315 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2023-11-29T08:50:54.315 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Ganglia.jpg 2023-11-29T08:50:54.315 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/OperationalModes.html 2023-11-29T08:50:54.315 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Playback.html 2023-11-29T08:50:54.315 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Hello.html 2023-11-29T08:50:54.315 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Sockets.html 2023-11-29T08:50:54.315 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Network.html 2023-11-29T08:50:54.315 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Infiniband.html 2023-11-29T08:50:54.315 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Memory.html 2023-11-29T08:50:54.315 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2023-11-29T08:50:54.315 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/Disks.html 2023-11-29T08:50:54.315 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/docs/NfsInfo.html 2023-11-29T08:50:54.315 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/service/ 2023-11-29T08:50:54.315 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/service/collectl.service 2023-11-29T08:50:54.315 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/initd/ 2023-11-29T08:50:54.316 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/initd/collectl-suse 2023-11-29T08:50:54.316 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/initd/collectl-debian 2023-11-29T08:50:54.316 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/initd/collectl-generic 2023-11-29T08:50:54.316 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/initd/collectl 2023-11-29T08:50:54.316 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/misc.ph 2023-11-29T08:50:54.316 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/gexpr.ph 2023-11-29T08:50:54.316 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/statsd.ph 2023-11-29T08:50:54.316 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/envrules.std 2023-11-29T08:50:54.316 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/hello.ph 2023-11-29T08:50:54.316 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/vmsum.ph 2023-11-29T08:50:54.316 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/proctree.ph 2023-11-29T08:50:54.316 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/README-WINDOWS 2023-11-29T08:50:54.316 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/formatit.ph 2023-11-29T08:50:54.318 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/colmux 2023-11-29T08:50:54.318 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/INSTALL 2023-11-29T08:50:54.318 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/graphite.ph 2023-11-29T08:50:54.318 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/RELEASE-collectl 2023-11-29T08:50:54.318 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/vmstat.ph 2023-11-29T08:50:54.318 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/README 2023-11-29T08:50:54.318 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/man1/ 2023-11-29T08:50:54.318 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/man1/collectl.1 2023-11-29T08:50:54.318 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/man1/colmux.1 2023-11-29T08:50:54.318 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/lexpr.ph 2023-11-29T08:50:54.318 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/client.pl 2023-11-29T08:50:54.318 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/collectl 2023-11-29T08:50:54.319 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/COPYING 2023-11-29T08:50:54.319 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/vnet.ph 2023-11-29T08:50:54.319 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/GPL 2023-11-29T08:50:54.319 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/UNINSTALL 2023-11-29T08:50:54.319 INFO:teuthology.orchestra.run.smithi031.stdout:collectl-4.3.1/ARTISTIC 2023-11-29T08:50:54.377 INFO:teuthology.orchestra.run.smithi031.stderr:./INSTALL: line 90: chkconfig: command not found 2023-11-29T08:50:54.379 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2023-11-29T08:50:54.984 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:05:00 ago on Wed 29 Nov 2023 08:45:54 AM UTC. 2023-11-29T08:50:55.088 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2023-11-29T08:50:55.091 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2023-11-29T08:50:55.149 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-11-29T08:50:55.150 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:50:55.150 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-11-29T08:50:55.150 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:50:55.151 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-11-29T08:50:55.151 INFO:teuthology.orchestra.run.smithi031.stdout: librbd-devel x86_64 2:18.0.0-7483.g412bfade.el9 ceph 23 k 2023-11-29T08:50:55.151 INFO:teuthology.orchestra.run.smithi031.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2023-11-29T08:50:55.151 INFO:teuthology.orchestra.run.smithi031.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2023-11-29T08:50:55.151 INFO:teuthology.orchestra.run.smithi031.stdout: perf x86_64 5.14.0-388.el9 appstream 7.6 M 2023-11-29T08:50:55.151 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-11-29T08:50:55.151 INFO:teuthology.orchestra.run.smithi031.stdout: libradospp-devel x86_64 2:18.0.0-7483.g412bfade.el9 ceph 30 k 2023-11-29T08:50:55.151 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:55.151 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-11-29T08:50:55.152 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-11-29T08:50:55.152 INFO:teuthology.orchestra.run.smithi031.stdout:Install 5 Packages 2023-11-29T08:50:55.152 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:50:55.153 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 7.7 M 2023-11-29T08:50:55.153 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 11 M 2023-11-29T08:50:55.153 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-11-29T08:50:58.245 INFO:teuthology.orchestra.run.smithi031.stdout:(1/5): libradospp-devel-18.0.0-7483.g412bfade.e 66 kB/s | 30 kB 00:00 2023-11-29T08:50:58.270 INFO:teuthology.orchestra.run.smithi031.stdout:(2/5): librbd-devel-18.0.0-7483.g412bfade.el9.x 48 kB/s | 23 kB 00:00 2023-11-29T08:50:58.495 INFO:teuthology.orchestra.run.smithi031.stdout:(3/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 58 kB/s | 13 kB 00:00 2023-11-29T08:50:58.554 INFO:teuthology.orchestra.run.smithi031.stdout:(4/5): pdsh-2.34-7.el9.x86_64.rpm 384 kB/s | 118 kB 00:00 2023-11-29T08:51:01.212 INFO:teuthology.orchestra.run.smithi031.stdout:(5/5): perf-5.14.0-388.el9.x86_64.rpm 2.2 MB/s | 7.6 MB 00:03 2023-11-29T08:51:01.215 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-11-29T08:51:01.216 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.3 MB/s | 7.7 MB 00:06 2023-11-29T08:51:01.299 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-11-29T08:51:01.325 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-11-29T08:51:01.325 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-11-29T08:51:01.543 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-11-29T08:51:01.544 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-11-29T08:51:01.897 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-11-29T08:51:01.973 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2023-11-29T08:51:02.011 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2023-11-29T08:51:02.042 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libradospp-devel-2:18.0.0-7483.g412bfade.el9.x86_64 3/5 2023-11-29T08:51:02.411 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librbd-devel-2:18.0.0-7483.g412bfade.el9.x86_64 4/5 2023-11-29T08:51:02.534 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : perf-5.14.0-388.el9.x86_64 5/5 2023-11-29T08:51:03.526 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: perf-5.14.0-388.el9.x86_64 5/5 2023-11-29T08:51:03.526 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libradospp-devel-2:18.0.0-7483.g412bfade.el9.x86_64 1/5 2023-11-29T08:51:03.526 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd-devel-2:18.0.0-7483.g412bfade.el9.x86_64 2/5 2023-11-29T08:51:03.526 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : perf-5.14.0-388.el9.x86_64 3/5 2023-11-29T08:51:03.526 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2023-11-29T08:51:03.907 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2023-11-29T08:51:03.907 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:51:03.907 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-11-29T08:51:03.907 INFO:teuthology.orchestra.run.smithi031.stdout: libradospp-devel-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:51:03.907 INFO:teuthology.orchestra.run.smithi031.stdout: librbd-devel-2:18.0.0-7483.g412bfade.el9.x86_64 2023-11-29T08:51:03.908 INFO:teuthology.orchestra.run.smithi031.stdout: pdsh-2.34-7.el9.x86_64 2023-11-29T08:51:03.908 INFO:teuthology.orchestra.run.smithi031.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2023-11-29T08:51:03.908 INFO:teuthology.orchestra.run.smithi031.stdout: perf-5.14.0-388.el9.x86_64 2023-11-29T08:51:03.908 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:51:03.908 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-11-29T08:51:04.330 INFO:tasks.cbt:benchmark: librbdfio 2023-11-29T08:51:04.331 DEBUG:teuthology.orchestra.run.smithi031:> git clone -b master https://github.com/axboe/fio.git /home/ubuntu/cephtest/fio 2023-11-29T08:51:04.347 INFO:teuthology.orchestra.run.smithi031.stderr:Cloning into '/home/ubuntu/cephtest/fio'... 2023-11-29T08:51:07.606 DEBUG:teuthology.orchestra.run.smithi031:> cd /home/ubuntu/cephtest/fio && ./configure && make 2023-11-29T08:51:08.155 INFO:teuthology.orchestra.run.smithi031.stdout:Operating system Linux 2023-11-29T08:51:08.155 INFO:teuthology.orchestra.run.smithi031.stdout:CPU x86_64 2023-11-29T08:51:08.155 INFO:teuthology.orchestra.run.smithi031.stdout:Big endian no 2023-11-29T08:51:08.155 INFO:teuthology.orchestra.run.smithi031.stdout:Compiler gcc 2023-11-29T08:51:08.155 INFO:teuthology.orchestra.run.smithi031.stdout:Cross compile no 2023-11-29T08:51:08.155 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:51:08.155 INFO:teuthology.orchestra.run.smithi031.stdout:Static build no 2023-11-29T08:51:08.267 INFO:teuthology.orchestra.run.smithi031.stdout:Wordsize 64 2023-11-29T08:51:08.275 INFO:teuthology.orchestra.run.smithi031.stdout:zlib no 2023-11-29T08:51:08.328 INFO:teuthology.orchestra.run.smithi031.stdout:fcntl(F_FULLFSYNC) no 2023-11-29T08:51:08.343 INFO:teuthology.orchestra.run.smithi031.stdout:Linux AIO support no 2023-11-29T08:51:08.343 INFO:teuthology.orchestra.run.smithi031.stdout:Linux AIO support rw flags no 2023-11-29T08:51:08.401 INFO:teuthology.orchestra.run.smithi031.stdout:POSIX AIO support yes 2023-11-29T08:51:08.402 INFO:teuthology.orchestra.run.smithi031.stdout:POSIX AIO support needs -lrt no 2023-11-29T08:51:08.433 INFO:teuthology.orchestra.run.smithi031.stdout:POSIX AIO fsync yes 2023-11-29T08:51:08.464 INFO:teuthology.orchestra.run.smithi031.stdout:POSIX pshared support yes 2023-11-29T08:51:08.511 INFO:teuthology.orchestra.run.smithi031.stdout:pthread_condattr_setclock() yes 2023-11-29T08:51:08.550 INFO:teuthology.orchestra.run.smithi031.stdout:pthread_sigmask() yes 2023-11-29T08:51:08.587 INFO:teuthology.orchestra.run.smithi031.stdout:pthread_getaffinity_np() yes 2023-11-29T08:51:08.597 INFO:teuthology.orchestra.run.smithi031.stdout:Solaris AIO support no 2023-11-29T08:51:08.643 INFO:teuthology.orchestra.run.smithi031.stdout:__sync_fetch_and_add yes 2023-11-29T08:51:08.691 INFO:teuthology.orchestra.run.smithi031.stdout:__sync_synchronize yes 2023-11-29T08:51:08.721 INFO:teuthology.orchestra.run.smithi031.stdout:__sync_val_compare_and_swap yes 2023-11-29T08:51:08.728 INFO:teuthology.orchestra.run.smithi031.stdout:libverbs no 2023-11-29T08:51:08.753 INFO:teuthology.orchestra.run.smithi031.stdout:rdmacm no 2023-11-29T08:51:08.760 INFO:teuthology.orchestra.run.smithi031.stdout:librpma no 2023-11-29T08:51:08.771 INFO:teuthology.orchestra.run.smithi031.stdout:libprotobuf_c no 2023-11-29T08:51:08.803 INFO:teuthology.orchestra.run.smithi031.stdout:asprintf() yes 2023-11-29T08:51:08.834 INFO:teuthology.orchestra.run.smithi031.stdout:vasprintf() yes 2023-11-29T08:51:08.866 INFO:teuthology.orchestra.run.smithi031.stdout:Linux fallocate yes 2023-11-29T08:51:08.899 INFO:teuthology.orchestra.run.smithi031.stdout:POSIX fadvise yes 2023-11-29T08:51:08.931 INFO:teuthology.orchestra.run.smithi031.stdout:POSIX fallocate yes 2023-11-29T08:51:08.962 INFO:teuthology.orchestra.run.smithi031.stdout:sched_setaffinity(3 arg) yes 2023-11-29T08:51:08.962 INFO:teuthology.orchestra.run.smithi031.stdout:sched_setaffinity(2 arg) no 2023-11-29T08:51:08.994 INFO:teuthology.orchestra.run.smithi031.stdout:clock_gettime yes 2023-11-29T08:51:09.028 INFO:teuthology.orchestra.run.smithi031.stdout:CLOCK_MONOTONIC yes 2023-11-29T08:51:09.072 INFO:teuthology.orchestra.run.smithi031.stdout:clockid_t yes 2023-11-29T08:51:09.107 INFO:teuthology.orchestra.run.smithi031.stdout:gettimeofday yes 2023-11-29T08:51:09.141 INFO:teuthology.orchestra.run.smithi031.stdout:fdatasync yes 2023-11-29T08:51:09.172 INFO:teuthology.orchestra.run.smithi031.stdout:pipe() yes 2023-11-29T08:51:09.204 INFO:teuthology.orchestra.run.smithi031.stdout:pipe2() yes 2023-11-29T08:51:09.235 INFO:teuthology.orchestra.run.smithi031.stdout:pread() yes 2023-11-29T08:51:09.313 INFO:teuthology.orchestra.run.smithi031.stdout:sync_file_range yes 2023-11-29T08:51:09.320 INFO:teuthology.orchestra.run.smithi031.stdout:ASharedMemory_create no 2023-11-29T08:51:09.347 INFO:teuthology.orchestra.run.smithi031.stdout:EXT4 move extent yes 2023-11-29T08:51:09.380 INFO:teuthology.orchestra.run.smithi031.stdout:Linux splice(2) yes 2023-11-29T08:51:09.387 INFO:teuthology.orchestra.run.smithi031.stdout:libnuma no 2023-11-29T08:51:09.419 INFO:teuthology.orchestra.run.smithi031.stdout:strsep yes 2023-11-29T08:51:09.449 INFO:teuthology.orchestra.run.smithi031.stdout:strcasestr yes 2023-11-29T08:51:09.464 INFO:teuthology.orchestra.run.smithi031.stdout:strlcat no 2023-11-29T08:51:09.498 INFO:teuthology.orchestra.run.smithi031.stdout:getopt_long_only() yes 2023-11-29T08:51:09.555 INFO:teuthology.orchestra.run.smithi031.stdout:inet_aton yes 2023-11-29T08:51:09.587 INFO:teuthology.orchestra.run.smithi031.stdout:socklen_t yes 2023-11-29T08:51:09.620 INFO:teuthology.orchestra.run.smithi031.stdout:__thread yes 2023-11-29T08:51:09.652 INFO:teuthology.orchestra.run.smithi031.stdout:RUSAGE_THREAD yes 2023-11-29T08:51:09.682 INFO:teuthology.orchestra.run.smithi031.stdout:SCHED_IDLE yes 2023-11-29T08:51:09.717 INFO:teuthology.orchestra.run.smithi031.stdout:TCP_NODELAY yes 2023-11-29T08:51:09.752 INFO:teuthology.orchestra.run.smithi031.stdout:Net engine window_size yes 2023-11-29T08:51:09.787 INFO:teuthology.orchestra.run.smithi031.stdout:TCP_MAXSEG yes 2023-11-29T08:51:09.817 INFO:teuthology.orchestra.run.smithi031.stdout:RLIMIT_MEMLOCK yes 2023-11-29T08:51:09.852 INFO:teuthology.orchestra.run.smithi031.stdout:pwritev/preadv yes 2023-11-29T08:51:09.885 INFO:teuthology.orchestra.run.smithi031.stdout:pwritev2/preadv2 yes 2023-11-29T08:51:09.923 INFO:teuthology.orchestra.run.smithi031.stdout:IPv6 helpers yes 2023-11-29T08:51:09.939 INFO:teuthology.orchestra.run.smithi031.stdout:http engine no 2023-11-29T08:51:10.036 INFO:teuthology.orchestra.run.smithi031.stdout:Rados engine yes 2023-11-29T08:51:10.143 INFO:teuthology.orchestra.run.smithi031.stdout:Rados Block Device engine yes 2023-11-29T08:51:10.263 INFO:teuthology.orchestra.run.smithi031.stdout:rbd_poll yes 2023-11-29T08:51:10.371 INFO:teuthology.orchestra.run.smithi031.stdout:rbd_invalidate_cache yes 2023-11-29T08:51:10.402 INFO:teuthology.orchestra.run.smithi031.stdout:setvbuf yes 2023-11-29T08:51:10.410 INFO:teuthology.orchestra.run.smithi031.stdout:Gluster API engine no 2023-11-29T08:51:10.426 INFO:teuthology.orchestra.run.smithi031.stdout:s390_z196_facilities no 2023-11-29T08:51:10.426 INFO:teuthology.orchestra.run.smithi031.stdout:HDFS engine no 2023-11-29T08:51:10.473 INFO:teuthology.orchestra.run.smithi031.stdout:MTD yes 2023-11-29T08:51:10.483 INFO:teuthology.orchestra.run.smithi031.stdout:libpmem no 2023-11-29T08:51:10.483 INFO:teuthology.orchestra.run.smithi031.stdout:libpmem1_5 no 2023-11-29T08:51:10.490 INFO:teuthology.orchestra.run.smithi031.stdout:libpmem2 no 2023-11-29T08:51:10.490 INFO:teuthology.orchestra.run.smithi031.stdout:PMDK dev-dax engine no 2023-11-29T08:51:10.490 INFO:teuthology.orchestra.run.smithi031.stdout:PMDK libpmem engine no 2023-11-29T08:51:10.498 INFO:teuthology.orchestra.run.smithi031.stdout:DDN's Infinite Memory Engine no 2023-11-29T08:51:10.498 INFO:teuthology.orchestra.run.smithi031.stdout:iscsi engine no 2023-11-29T08:51:10.498 INFO:teuthology.orchestra.run.smithi031.stdout:NBD engine no 2023-11-29T08:51:10.507 INFO:teuthology.orchestra.run.smithi031.stdout:DAOS File System (dfs) Engine no 2023-11-29T08:51:10.517 INFO:teuthology.orchestra.run.smithi031.stdout:NFS engine no 2023-11-29T08:51:10.517 INFO:teuthology.orchestra.run.smithi031.stdout:lex/yacc for arithmetic no 2023-11-29T08:51:10.557 INFO:teuthology.orchestra.run.smithi031.stdout:getmntent yes 2023-11-29T08:51:10.588 INFO:teuthology.orchestra.run.smithi031.stdout:getmntinfo no 2023-11-29T08:51:10.645 INFO:teuthology.orchestra.run.smithi031.stdout:Static Assert yes 2023-11-29T08:51:10.673 INFO:teuthology.orchestra.run.smithi031.stdout:bool yes 2023-11-29T08:51:10.707 INFO:teuthology.orchestra.run.smithi031.stdout:strndup yes 2023-11-29T08:51:10.715 INFO:teuthology.orchestra.run.smithi031.stdout:Valgrind headers no 2023-11-29T08:51:10.759 INFO:teuthology.orchestra.run.smithi031.stdout:Zoned block device support yes 2023-11-29T08:51:10.788 INFO:teuthology.orchestra.run.smithi031.stdout:Zoned block device capacity yes 2023-11-29T08:51:10.795 INFO:teuthology.orchestra.run.smithi031.stdout:libzbc engine no 2023-11-29T08:51:10.838 INFO:teuthology.orchestra.run.smithi031.stdout:NVMe uring command support yes 2023-11-29T08:51:10.853 INFO:teuthology.orchestra.run.smithi031.stdout:xnvme engine no 2023-11-29T08:51:10.860 INFO:teuthology.orchestra.run.smithi031.stdout:isal no 2023-11-29T08:51:10.875 INFO:teuthology.orchestra.run.smithi031.stdout:libblkio engine no 2023-11-29T08:51:10.876 INFO:teuthology.orchestra.run.smithi031.stdout:march_armv8_a_crc_crypto no 2023-11-29T08:51:10.876 INFO:teuthology.orchestra.run.smithi031.stdout:cuda no 2023-11-29T08:51:10.876 INFO:teuthology.orchestra.run.smithi031.stdout:libcufile no 2023-11-29T08:51:10.905 INFO:teuthology.orchestra.run.smithi031.stdout:Build march=native yes 2023-11-29T08:51:10.912 INFO:teuthology.orchestra.run.smithi031.stdout:CUnit no 2023-11-29T08:51:10.941 INFO:teuthology.orchestra.run.smithi031.stdout:__kernel_rwf_t yes 2023-11-29T08:51:10.969 INFO:teuthology.orchestra.run.smithi031.stdout:-Wimplicit-fallthrough=2 yes 2023-11-29T08:51:11.000 INFO:teuthology.orchestra.run.smithi031.stdout:-Wno-stringop-truncation yes 2023-11-29T08:51:11.032 INFO:teuthology.orchestra.run.smithi031.stdout:MADV_HUGEPAGE yes 2023-11-29T08:51:11.063 INFO:teuthology.orchestra.run.smithi031.stdout:gettid yes 2023-11-29T08:51:11.098 INFO:teuthology.orchestra.run.smithi031.stdout:statx(2)/libc yes 2023-11-29T08:51:11.133 INFO:teuthology.orchestra.run.smithi031.stdout:statx(2)/syscall yes 2023-11-29T08:51:11.135 INFO:teuthology.orchestra.run.smithi031.stdout:Windows PDB generation no 2023-11-29T08:51:11.167 INFO:teuthology.orchestra.run.smithi031.stdout:timerfd_create yes 2023-11-29T08:51:11.169 INFO:teuthology.orchestra.run.smithi031.stdout:Consider installing zlib1g-dev (zlib-devel) as some fio features depend on it. 2023-11-29T08:51:11.169 INFO:teuthology.orchestra.run.smithi031.stdout:Lib-based ioengines dynamic no 2023-11-29T08:51:11.233 INFO:teuthology.orchestra.run.smithi031.stdout:TCMalloc support yes 2023-11-29T08:51:11.234 INFO:teuthology.orchestra.run.smithi031.stdout:seed_buckets 4 2023-11-29T08:51:11.325 INFO:teuthology.orchestra.run.smithi031.stderr:FIO_VERSION = fio-3.36-19-gbdf99 2023-11-29T08:51:11.359 INFO:teuthology.orchestra.run.smithi031.stdout: CC crc/crc16.o 2023-11-29T08:51:11.452 INFO:teuthology.orchestra.run.smithi031.stdout: CC crc/crc32.o 2023-11-29T08:51:11.490 INFO:teuthology.orchestra.run.smithi031.stdout: CC crc/crc32c-arm64.o 2023-11-29T08:51:11.564 INFO:teuthology.orchestra.run.smithi031.stdout: CC crc/crc32c-intel.o 2023-11-29T08:51:11.610 INFO:teuthology.orchestra.run.smithi031.stdout: CC crc/crc32c.o 2023-11-29T08:51:11.652 INFO:teuthology.orchestra.run.smithi031.stdout: CC crc/crc64.o 2023-11-29T08:51:11.692 INFO:teuthology.orchestra.run.smithi031.stdout: CC crc/crc7.o 2023-11-29T08:51:11.726 INFO:teuthology.orchestra.run.smithi031.stdout: CC crc/crct10dif_common.o 2023-11-29T08:51:11.760 INFO:teuthology.orchestra.run.smithi031.stdout: CC crc/fnv.o 2023-11-29T08:51:11.801 INFO:teuthology.orchestra.run.smithi031.stdout: CC crc/md5.o 2023-11-29T08:51:11.913 INFO:teuthology.orchestra.run.smithi031.stdout: CC crc/murmur3.o 2023-11-29T08:51:11.958 INFO:teuthology.orchestra.run.smithi031.stdout: CC crc/sha1.o 2023-11-29T08:51:12.155 INFO:teuthology.orchestra.run.smithi031.stdout: CC crc/sha256.o 2023-11-29T08:51:12.444 INFO:teuthology.orchestra.run.smithi031.stdout: CC crc/sha3.o 2023-11-29T08:51:12.591 INFO:teuthology.orchestra.run.smithi031.stdout: CC crc/sha512.o 2023-11-29T08:51:12.714 INFO:teuthology.orchestra.run.smithi031.stdout: CC crc/test.o 2023-11-29T08:51:12.880 INFO:teuthology.orchestra.run.smithi031.stdout: CC crc/xxhash.o 2023-11-29T08:51:12.977 INFO:teuthology.orchestra.run.smithi031.stdout: CC lib/axmap.o 2023-11-29T08:51:13.104 INFO:teuthology.orchestra.run.smithi031.stdout: CC lib/bloom.o 2023-11-29T08:51:13.189 INFO:teuthology.orchestra.run.smithi031.stdout: CC lib/flist_sort.o 2023-11-29T08:51:13.262 INFO:teuthology.orchestra.run.smithi031.stdout: CC lib/gauss.o 2023-11-29T08:51:13.364 INFO:teuthology.orchestra.run.smithi031.stdout: CC lib/getrusage.o 2023-11-29T08:51:13.401 INFO:teuthology.orchestra.run.smithi031.stdout: CC lib/hweight.o 2023-11-29T08:51:13.441 INFO:teuthology.orchestra.run.smithi031.stdout: CC lib/ieee754.o 2023-11-29T08:51:13.493 INFO:teuthology.orchestra.run.smithi031.stdout: CC lib/lfsr.o 2023-11-29T08:51:13.596 INFO:teuthology.orchestra.run.smithi031.stdout: CC lib/memalign.o 2023-11-29T08:51:13.640 INFO:teuthology.orchestra.run.smithi031.stdout: CC lib/memcpy.o 2023-11-29T08:51:13.788 INFO:teuthology.orchestra.run.smithi031.stdout: CC lib/mountcheck.o 2023-11-29T08:51:13.830 INFO:teuthology.orchestra.run.smithi031.stdout: CC lib/num2str.o 2023-11-29T08:51:13.897 INFO:teuthology.orchestra.run.smithi031.stdout: CC lib/output_buffer.o 2023-11-29T08:51:13.945 INFO:teuthology.orchestra.run.smithi031.stdout: CC lib/pattern.o 2023-11-29T08:51:14.110 INFO:teuthology.orchestra.run.smithi031.stdout: CC lib/prio_tree.o 2023-11-29T08:51:14.234 INFO:teuthology.orchestra.run.smithi031.stdout: CC lib/rand.o 2023-11-29T08:51:14.428 INFO:teuthology.orchestra.run.smithi031.stdout: CC lib/rbtree.o 2023-11-29T08:51:14.555 INFO:teuthology.orchestra.run.smithi031.stdout: CC lib/strntol.o 2023-11-29T08:51:14.604 INFO:teuthology.orchestra.run.smithi031.stdout: CC lib/zipf.o 2023-11-29T08:51:14.728 INFO:teuthology.orchestra.run.smithi031.stdout: CC gettime.o 2023-11-29T08:51:15.028 INFO:teuthology.orchestra.run.smithi031.stdout: CC ioengines.o 2023-11-29T08:51:15.305 INFO:teuthology.orchestra.run.smithi031.stdout: CC init.o 2023-11-29T08:51:16.085 INFO:teuthology.orchestra.run.smithi031.stdout: CC stat.o 2023-11-29T08:51:18.032 INFO:teuthology.orchestra.run.smithi031.stdout: CC log.o 2023-11-29T08:51:18.159 INFO:teuthology.orchestra.run.smithi031.stdout: CC time.o 2023-11-29T08:51:18.288 INFO:teuthology.orchestra.run.smithi031.stdout: CC filesetup.o 2023-11-29T08:51:19.008 INFO:teuthology.orchestra.run.smithi031.stdout: CC eta.o 2023-11-29T08:51:19.334 INFO:teuthology.orchestra.run.smithi031.stdout: CC verify.o 2023-11-29T08:51:20.040 INFO:teuthology.orchestra.run.smithi031.stdout: CC memory.o 2023-11-29T08:51:20.220 INFO:teuthology.orchestra.run.smithi031.stdout: CC io_u.o 2023-11-29T08:51:21.000 INFO:teuthology.orchestra.run.smithi031.stdout: CC parse.o 2023-11-29T08:51:21.456 INFO:teuthology.orchestra.run.smithi031.stdout: CC fio_sem.o 2023-11-29T08:51:21.564 INFO:teuthology.orchestra.run.smithi031.stdout: CC rwlock.o 2023-11-29T08:51:21.648 INFO:teuthology.orchestra.run.smithi031.stdout: CC pshared.o 2023-11-29T08:51:21.717 INFO:teuthology.orchestra.run.smithi031.stdout: CC options.o 2023-11-29T08:51:22.530 INFO:teuthology.orchestra.run.smithi031.stdout: CC smalloc.o 2023-11-29T08:51:22.725 INFO:teuthology.orchestra.run.smithi031.stdout: CC filehash.o 2023-11-29T08:51:22.878 INFO:teuthology.orchestra.run.smithi031.stdout: CC profile.o 2023-11-29T08:51:23.001 INFO:teuthology.orchestra.run.smithi031.stdout: CC debug.o 2023-11-29T08:51:23.042 INFO:teuthology.orchestra.run.smithi031.stdout: CC engines/cpu.o 2023-11-29T08:51:23.189 INFO:teuthology.orchestra.run.smithi031.stdout: CC engines/mmap.o 2023-11-29T08:51:23.359 INFO:teuthology.orchestra.run.smithi031.stdout: CC engines/sync.o 2023-11-29T08:51:23.598 INFO:teuthology.orchestra.run.smithi031.stdout: CC engines/null.o 2023-11-29T08:51:23.716 INFO:teuthology.orchestra.run.smithi031.stdout: CC engines/net.o 2023-11-29T08:51:24.139 INFO:teuthology.orchestra.run.smithi031.stdout: CC engines/ftruncate.o 2023-11-29T08:51:24.237 INFO:teuthology.orchestra.run.smithi031.stdout: CC engines/fileoperations.o 2023-11-29T08:51:24.384 INFO:teuthology.orchestra.run.smithi031.stdout: CC engines/exec.o 2023-11-29T08:51:24.539 INFO:teuthology.orchestra.run.smithi031.stdout: CC server.o 2023-11-29T08:51:25.309 INFO:teuthology.orchestra.run.smithi031.stdout: CC client.o 2023-11-29T08:51:25.869 INFO:teuthology.orchestra.run.smithi031.stdout: CC iolog.o 2023-11-29T08:51:26.331 INFO:teuthology.orchestra.run.smithi031.stdout: CC backend.o 2023-11-29T08:51:27.201 INFO:teuthology.orchestra.run.smithi031.stdout: CC libfio.o 2023-11-29T08:51:27.386 INFO:teuthology.orchestra.run.smithi031.stdout: CC flow.o 2023-11-29T08:51:27.508 INFO:teuthology.orchestra.run.smithi031.stdout: CC cconv.o 2023-11-29T08:51:27.882 INFO:teuthology.orchestra.run.smithi031.stdout: CC gettime-thread.o 2023-11-29T08:51:27.986 INFO:teuthology.orchestra.run.smithi031.stdout: CC helpers.o 2023-11-29T08:51:28.050 INFO:teuthology.orchestra.run.smithi031.stdout: CC json.o 2023-11-29T08:51:28.239 INFO:teuthology.orchestra.run.smithi031.stdout: CC idletime.o 2023-11-29T08:51:28.469 INFO:teuthology.orchestra.run.smithi031.stdout: CC td_error.o 2023-11-29T08:51:28.572 INFO:teuthology.orchestra.run.smithi031.stdout: CC profiles/tiobench.o 2023-11-29T08:51:28.673 INFO:teuthology.orchestra.run.smithi031.stdout: CC profiles/act.o 2023-11-29T08:51:28.839 INFO:teuthology.orchestra.run.smithi031.stdout: CC io_u_queue.o 2023-11-29T08:51:28.890 INFO:teuthology.orchestra.run.smithi031.stdout: CC filelock.o 2023-11-29T08:51:29.023 INFO:teuthology.orchestra.run.smithi031.stdout: CC workqueue.o 2023-11-29T08:51:29.202 INFO:teuthology.orchestra.run.smithi031.stdout: CC rate-submit.o 2023-11-29T08:51:29.354 INFO:teuthology.orchestra.run.smithi031.stdout: CC optgroup.o 2023-11-29T08:51:29.400 INFO:teuthology.orchestra.run.smithi031.stdout: CC helper_thread.o 2023-11-29T08:51:29.583 INFO:teuthology.orchestra.run.smithi031.stdout: CC steadystate.o 2023-11-29T08:51:29.833 INFO:teuthology.orchestra.run.smithi031.stdout: CC zone-dist.o 2023-11-29T08:51:29.968 INFO:teuthology.orchestra.run.smithi031.stdout: CC zbd.o 2023-11-29T08:51:30.687 INFO:teuthology.orchestra.run.smithi031.stdout: CC dedupe.o 2023-11-29T08:51:30.840 INFO:teuthology.orchestra.run.smithi031.stdout: CC fdp.o 2023-11-29T08:51:30.988 INFO:teuthology.orchestra.run.smithi031.stdout: CC engines/posixaio.o 2023-11-29T08:51:31.126 INFO:teuthology.orchestra.run.smithi031.stdout: CC engines/falloc.o 2023-11-29T08:51:31.241 INFO:teuthology.orchestra.run.smithi031.stdout: CC engines/e4defrag.o 2023-11-29T08:51:31.378 INFO:teuthology.orchestra.run.smithi031.stdout: CC engines/splice.o 2023-11-29T08:51:31.575 INFO:teuthology.orchestra.run.smithi031.stdout: CC oslib/asprintf.o 2023-11-29T08:51:31.611 INFO:teuthology.orchestra.run.smithi031.stdout: CC oslib/strlcat.o 2023-11-29T08:51:31.656 INFO:teuthology.orchestra.run.smithi031.stdout: CC engines/mtd.o 2023-11-29T08:51:31.823 INFO:teuthology.orchestra.run.smithi031.stdout: CC oslib/libmtd.o 2023-11-29T08:51:32.221 INFO:teuthology.orchestra.run.smithi031.stdout: CC oslib/libmtd_legacy.o 2023-11-29T08:51:32.358 INFO:teuthology.orchestra.run.smithi031.stdout: CC diskutil.o 2023-11-29T08:51:32.574 INFO:teuthology.orchestra.run.smithi031.stdout: CC fifo.o 2023-11-29T08:51:32.631 INFO:teuthology.orchestra.run.smithi031.stdout: CC blktrace.o 2023-11-29T08:51:32.976 INFO:teuthology.orchestra.run.smithi031.stdout: CC cgroup.o 2023-11-29T08:51:33.127 INFO:teuthology.orchestra.run.smithi031.stdout: CC trim.o 2023-11-29T08:51:33.244 INFO:teuthology.orchestra.run.smithi031.stdout: CC engines/sg.o 2023-11-29T08:51:33.641 INFO:teuthology.orchestra.run.smithi031.stdout: CC oslib/linux-dev-lookup.o 2023-11-29T08:51:33.700 INFO:teuthology.orchestra.run.smithi031.stdout: CC engines/io_uring.o 2023-11-29T08:51:34.094 INFO:teuthology.orchestra.run.smithi031.stdout: CC engines/nvme.o 2023-11-29T08:51:34.391 INFO:teuthology.orchestra.run.smithi031.stdout: CC oslib/linux-blkzoned.o 2023-11-29T08:51:34.547 INFO:teuthology.orchestra.run.smithi031.stdout: CC engines/cmdprio.o 2023-11-29T08:51:34.797 INFO:teuthology.orchestra.run.smithi031.stdout: CC engines/rados.o 2023-11-29T08:51:34.973 INFO:teuthology.orchestra.run.smithi031.stdout: CC engines/rbd.o 2023-11-29T08:51:35.160 INFO:teuthology.orchestra.run.smithi031.stdout: CC fio.o 2023-11-29T08:51:35.251 INFO:teuthology.orchestra.run.smithi031.stdout: LINK fio 2023-11-29T08:51:35.451 INFO:teuthology.orchestra.run.smithi031.stdout: CC t/log.o 2023-11-29T08:51:35.493 INFO:teuthology.orchestra.run.smithi031.stdout: CC t/genzipf.o 2023-11-29T08:51:35.642 INFO:teuthology.orchestra.run.smithi031.stdout: CC oslib/strcasestr.o 2023-11-29T08:51:35.666 INFO:teuthology.orchestra.run.smithi031.stdout: CC oslib/strndup.o 2023-11-29T08:51:35.690 INFO:teuthology.orchestra.run.smithi031.stdout: LINK t/fio-genzipf 2023-11-29T08:51:35.783 INFO:teuthology.orchestra.run.smithi031.stdout: CC t/btrace2fio.o 2023-11-29T08:51:36.211 INFO:teuthology.orchestra.run.smithi031.stdout: LINK t/fio-btrace2fio 2023-11-29T08:51:36.303 INFO:teuthology.orchestra.run.smithi031.stdout: CC t/verify-state.o 2023-11-29T08:51:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: CC t/debug.o 2023-11-29T08:51:36.443 INFO:teuthology.orchestra.run.smithi031.stdout: LINK t/fio-verify-state 2023-11-29T08:51:36.538 INFO:teuthology.orchestra.run.smithi031.stdout: CC t/stest.o 2023-11-29T08:51:36.609 INFO:teuthology.orchestra.run.smithi031.stdout: CC t/arch.o 2023-11-29T08:51:36.639 INFO:teuthology.orchestra.run.smithi031.stdout: LINK t/stest 2023-11-29T08:51:36.735 INFO:teuthology.orchestra.run.smithi031.stdout: CC t/ieee754.o 2023-11-29T08:51:36.776 INFO:teuthology.orchestra.run.smithi031.stdout: LINK t/ieee754 2023-11-29T08:51:36.867 INFO:teuthology.orchestra.run.smithi031.stdout: CC t/axmap.o 2023-11-29T08:51:36.968 INFO:teuthology.orchestra.run.smithi031.stdout: LINK t/axmap 2023-11-29T08:51:37.059 INFO:teuthology.orchestra.run.smithi031.stdout: CC t/lfsr-test.o 2023-11-29T08:51:37.160 INFO:teuthology.orchestra.run.smithi031.stdout: LINK t/lfsr-test 2023-11-29T08:51:37.252 INFO:teuthology.orchestra.run.smithi031.stdout: CC t/gen-rand.o 2023-11-29T08:51:37.352 INFO:teuthology.orchestra.run.smithi031.stdout: LINK t/gen-rand 2023-11-29T08:51:37.444 INFO:teuthology.orchestra.run.smithi031.stdout: CC t/memlock.o 2023-11-29T08:51:37.507 INFO:teuthology.orchestra.run.smithi031.stdout: LINK t/memlock 2023-11-29T08:51:37.601 INFO:teuthology.orchestra.run.smithi031.stdout: CC t/read-to-pipe-async.o 2023-11-29T08:51:37.797 INFO:teuthology.orchestra.run.smithi031.stdout: LINK t/read-to-pipe-async 2023-11-29T08:51:37.888 INFO:teuthology.orchestra.run.smithi031.stdout: CC t/io_uring.o 2023-11-29T08:51:38.445 INFO:teuthology.orchestra.run.smithi031.stdout: LINK t/io_uring 2023-11-29T08:51:38.543 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2023-11-29T08:51:38.575 DEBUG:teuthology.orchestra.run.smithi031:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-11-29T08:51:38.837 INFO:teuthology.orchestra.run.smithi031.stderr:08:51:38 - DEBUG - cbt - Settings.cluster: 2023-11-29T08:51:38.837 INFO:teuthology.orchestra.run.smithi031.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2023-11-29T08:51:38.838 INFO:teuthology.orchestra.run.smithi031.stderr: 'clients': ['ubuntu@smithi031.front.sepia.ceph.com'], 2023-11-29T08:51:38.838 INFO:teuthology.orchestra.run.smithi031.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2023-11-29T08:51:38.838 INFO:teuthology.orchestra.run.smithi031.stderr: 'head': 'ubuntu@smithi031.front.sepia.ceph.com', 2023-11-29T08:51:38.838 INFO:teuthology.orchestra.run.smithi031.stderr: 'is_teuthology': True, 2023-11-29T08:51:38.838 INFO:teuthology.orchestra.run.smithi031.stderr: 'iterations': 1, 2023-11-29T08:51:38.838 INFO:teuthology.orchestra.run.smithi031.stderr: 'mons': ['ubuntu@smithi031.front.sepia.ceph.com'], 2023-11-29T08:51:38.838 INFO:teuthology.orchestra.run.smithi031.stderr: 'osds': ['ubuntu@smithi031.front.sepia.ceph.com'], 2023-11-29T08:51:38.838 INFO:teuthology.orchestra.run.smithi031.stderr: 'osds_per_node': 3, 2023-11-29T08:51:38.838 INFO:teuthology.orchestra.run.smithi031.stderr: 'pid_dir': '/var/run/ceph', 2023-11-29T08:51:38.838 INFO:teuthology.orchestra.run.smithi031.stderr: 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 2023-11-29T08:51:38.838 INFO:teuthology.orchestra.run.smithi031.stderr: 'rebuild_every_test': False, 2023-11-29T08:51:38.838 INFO:teuthology.orchestra.run.smithi031.stderr: 'rgws': {'ubuntu@smithi031.front.sepia.ceph.com': None}, 2023-11-29T08:51:38.838 INFO:teuthology.orchestra.run.smithi031.stderr: 'tmp_dir': '/tmp/cbt', 2023-11-29T08:51:38.838 INFO:teuthology.orchestra.run.smithi031.stderr: 'use_existing': True, 2023-11-29T08:51:38.839 INFO:teuthology.orchestra.run.smithi031.stderr: 'user': 'ubuntu'} 2023-11-29T08:51:38.839 INFO:teuthology.orchestra.run.smithi031.stderr:08:51:38 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:51:38.839 INFO:teuthology.orchestra.run.smithi031.stderr:08:51:38 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:51:38.839 INFO:teuthology.orchestra.run.smithi031.stderr:08:51:38 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:51:38.839 INFO:teuthology.orchestra.run.smithi031.stderr:08:51:38 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite; fi 2023-11-29T08:51:38.842 INFO:teuthology.orchestra.run.smithi031.stderr:08:51:38 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:51:38.842 INFO:teuthology.orchestra.run.smithi031.stderr:08:51:38 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:51:38.842 INFO:teuthology.orchestra.run.smithi031.stderr:08:51:38 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite 2023-11-29T08:51:38.844 INFO:teuthology.orchestra.run.smithi031.stderr:08:51:38 - INFO - cbt - Pausing for 60s for idle monitoring. 2023-11-29T08:51:38.845 INFO:teuthology.orchestra.run.smithi031.stderr:08:51:38 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:51:38.845 INFO:teuthology.orchestra.run.smithi031.stderr:08:51:38 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:51:38.845 INFO:teuthology.orchestra.run.smithi031.stderr:08:51:38 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/idle_monitoring/collectl 2023-11-29T08:51:38.847 INFO:teuthology.orchestra.run.smithi031.stderr:08:51:38 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:51:38.847 INFO:teuthology.orchestra.run.smithi031.stderr:08:51:38 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-11-29T08:51:38.849 INFO:teuthology.orchestra.run.smithi031.stderr:08:51:38 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:51:38.849 INFO:teuthology.orchestra.run.smithi031.stderr:08:51:38 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:51:38.849 INFO:teuthology.orchestra.run.smithi031.stderr:08:51:38 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/idle_monitoring/perf 2023-11-29T08:51:38.852 INFO:teuthology.orchestra.run.smithi031.stderr:08:51:38 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:51:38.852 INFO:teuthology.orchestra.run.smithi031.stderr:08:51:38 - DEBUG - cbt - PerfMonitoring: in local_node 2023-11-29T08:51:38.852 INFO:teuthology.orchestra.run.smithi031.stderr:08:51:38 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-11-29T08:52:38.914 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:38 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:52:38.914 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:38 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:52:38.914 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:38 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-11-29T08:52:38.949 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:38 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:52:38.950 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:38 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:52:38.950 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:38 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-11-29T08:52:38.990 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:38 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:52:38.990 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:38 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:52:38.990 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:38 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/* 2023-11-29T08:52:39.011 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:39 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:52:39.012 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:39 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-7d4bb5d5/$(basename ${i}).smithi031.front.sepia.ceph.com ; done 2023-11-29T08:52:39.016 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:39 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:52:39.016 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:39 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:52:39.016 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/pool_monitoring/collectl 2023-11-29T08:52:39.019 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:39 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:52:39.019 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-11-29T08:52:39.021 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:39 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:52:39.021 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:39 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:52:39.021 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/pool_monitoring/perf 2023-11-29T08:52:39.024 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:39 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:52:39.024 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:39 - DEBUG - cbt - PerfMonitoring: in local_node 2023-11-29T08:52:39.024 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:39 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-11-29T08:52:39.024 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:39 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:52:39.024 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:39 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:52:39.024 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:39 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete cbt-librbdfio cbt-librbdfio --yes-i-really-really-mean-it 2023-11-29T08:52:39.332 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:39 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:52:39.332 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:39 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:52:39.332 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:39 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create cbt-librbdfio 128 128 2023-11-29T08:52:40.014 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:40 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:52:40.015 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:40 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:52:40.015 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:40 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable cbt-librbdfio rbd 2023-11-29T08:52:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:41 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:52:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:41 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:52:41.015 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:41 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio size 3 2023-11-29T08:52:42.016 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:42 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:52:42.016 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:42 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:52:42.016 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:42 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio min_size 2 2023-11-29T08:52:43.016 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:43 - INFO - cbt - Checking Health after pool creation. 2023-11-29T08:52:48.022 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:48 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:52:48.022 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:48 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:52:48.022 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:48 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-11-29T08:52:48.353 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:48 - INFO - cbt - Final Pool Health Check. 2023-11-29T08:52:53.362 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:53 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:52:53.362 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:53 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:52:53.362 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-11-29T08:52:53.673 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:53 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:52:53.673 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:53 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:52:53.673 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:53 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=hostname -f 2023-11-29T08:52:53.676 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:53 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:52:53.676 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:53 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:52:53.676 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:53 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi031.front.sepia.ceph.com-0 --size 4096 --pool cbt-librbdfio --order 22 2023-11-29T08:52:55.420 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:55 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:52:55.420 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:55 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:52:55.420 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:55 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi031.front.sepia.ceph.com-1 --size 4096 --pool cbt-librbdfio --order 22 2023-11-29T08:52:55.522 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:55 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:52:55.522 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:55 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:52:55.523 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-11-29T08:52:55.540 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:55 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:52:55.540 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:55 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:52:55.540 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-11-29T08:52:55.574 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:55 - INFO - cbt - Attempting to populating fio files... 2023-11-29T08:52:55.574 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:55 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:52:55.574 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:55 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:52:55.574 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2023-11-29T08:52:55.576 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:55 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:52:55.576 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:55 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:52:55.576 INFO:teuthology.orchestra.run.smithi031.stderr:08:52:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2023-11-29T08:55:30.329 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:30 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:55:30.329 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:30 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-7d4bb5d5. 2023-11-29T08:55:30.329 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:30 - INFO - cbt - Setting OSD Read Ahead to: 4096 2023-11-29T08:55:30.329 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:30 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:55:30.330 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:30 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:55:30.330 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 4096 > /sys/block/'{}'/queue/read_ahead_kb" 2023-11-29T08:55:30.331 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:30 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite 2023-11-29T08:55:30.331 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:30 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:55:30.331 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:30 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:55:30.331 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite 2023-11-29T08:55:30.352 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:30 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:55:30.352 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:30 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:55:30.352 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2023-11-29T08:55:31.292 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:31 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:55:31.292 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:31 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2023-11-29T08:55:32.078 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:32 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:55:32.078 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:32 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:55:32.078 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:32 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite 2023-11-29T08:55:32.189 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:32 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:55:32.189 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:32 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:55:32.189 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:32 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/ceph_settings.out 2023-11-29T08:55:58.151 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:58 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:55:58.151 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:58 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:55:58.151 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo "[`date`] ceph progress" >> /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/pgautoscaler.log;/usr/bin/ceph -c /etc/ceph/ceph.conf progress | tee -a /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/pgautoscaler.log 2023-11-29T08:55:58.505 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:58 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:55:58.505 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:58 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:55:58.506 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/collectl 2023-11-29T08:55:58.508 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:58 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:55:58.508 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-11-29T08:55:58.510 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:58 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:55:58.510 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:58 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:55:58.510 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/perf 2023-11-29T08:55:58.513 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:58 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:55:58.513 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:58 - DEBUG - cbt - PerfMonitoring: in local_node 2023-11-29T08:55:58.513 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:58 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-11-29T08:55:58.513 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:58 - INFO - cbt - Running rbd fio randwrite test. 2023-11-29T08:55:58.513 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:58 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-0 2023-11-29T08:55:58.514 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:58 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:55:58.514 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:58 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:55:58.514 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=randwrite --output-format=json,normal --runtime=60 --numjobs=1 --direct=1 --bs=4194304B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/output.0 2023-11-29T08:55:58.515 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:58 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-1 2023-11-29T08:55:58.515 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:58 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:55:58.515 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:58 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:55:58.515 INFO:teuthology.orchestra.run.smithi031.stderr:08:55:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=randwrite --output-format=json,normal --runtime=60 --numjobs=1 --direct=1 --bs=4194304B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/output.1 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/output.1 2023-11-29T08:57:01.858 INFO:teuthology.orchestra.run.smithi031.stderr:08:57:01 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:57:01.858 INFO:teuthology.orchestra.run.smithi031.stderr:08:57:01 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:57:01.858 INFO:teuthology.orchestra.run.smithi031.stderr:08:57:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-11-29T08:57:01.904 INFO:teuthology.orchestra.run.smithi031.stderr:08:57:01 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:57:01.904 INFO:teuthology.orchestra.run.smithi031.stderr:08:57:01 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:57:01.904 INFO:teuthology.orchestra.run.smithi031.stderr:08:57:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-11-29T08:57:01.937 INFO:teuthology.orchestra.run.smithi031.stderr:08:57:01 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:57:01.937 INFO:teuthology.orchestra.run.smithi031.stderr:08:57:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/perf/perf.data 2023-11-29T08:57:01.939 INFO:teuthology.orchestra.run.smithi031.stderr:08:57:01 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:57:01.939 INFO:teuthology.orchestra.run.smithi031.stderr:08:57:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/perf/perf_stat.* 2023-11-29T08:57:01.941 INFO:teuthology.orchestra.run.smithi031.stderr:08:57:01 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:57:01.941 INFO:teuthology.orchestra.run.smithi031.stderr:08:57:01 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:57:01.941 INFO:teuthology.orchestra.run.smithi031.stderr:08:57:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/historic_ops.out 2023-11-29T08:57:02.254 INFO:teuthology.orchestra.run.smithi031.stderr:08:57:02 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:57:02.273 INFO:teuthology.orchestra.run.smithi031.stderr:08:57:02 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:57:02.273 INFO:teuthology.orchestra.run.smithi031.stderr:08:57:02 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/* 2023-11-29T08:57:02.293 INFO:teuthology.orchestra.run.smithi031.stderr:08:57:02 - DEBUG - cbt - get_fqdn_local()=smithi031.front.sepia.ceph.com 2023-11-29T08:57:02.293 INFO:teuthology.orchestra.run.smithi031.stderr:08:57:02 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-7d4bb5d5/$(basename ${i}).smithi031.front.sepia.ceph.com ; done 2023-11-29T08:57:02.340 INFO:teuthology.orchestra.run.smithi031.stderr:08:57:02 - INFO - cbt - Convert results to json format. 2023-11-29T08:57:02.340 INFO:teuthology.orchestra.run.smithi031.stderr:08:57:02 - DEBUG - cbt - Nodes : ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:57:02.341 INFO:teuthology.orchestra.run.smithi031.stdout:cleaning remote dir /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randwrite 2023-11-29T08:57:02.341 INFO:teuthology.orchestra.run.smithi031.stdout:smithi031.front.sepia.ceph.com: smithi031.front.sepia.ceph.com 2023-11-29T08:57:02.341 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-11-29T08:57:02.341 INFO:teuthology.orchestra.run.smithi031.stdout:['smithi031.front.sepia.ceph.com'] 2023-11-29T08:57:02.356 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2023-11-29T08:57:02.365 DEBUG:teuthology.orchestra.run.smithi031:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2023-11-29T08:57:02.555 DEBUG:teuthology.orchestra.run.smithi031:> rm --one-file-system -rf -- /home/ubuntu/cephtest/fio 2023-11-29T08:57:02.611 INFO:tasks.cbt_performance:Collecting CBT performance data config 2023-11-29T08:57:02.611 INFO:tasks.cbt_performance:reading cbt preferences 2023-11-29T08:57:02.611 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -name 'json_output.*' 2023-11-29T08:57:02.692 INFO:teuthology.orchestra.run.smithi031.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-7d4bb5d5/json_output.1.smithi031.front.sepia.ceph.com 2023-11-29T08:57:02.692 INFO:teuthology.orchestra.run.smithi031.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-7d4bb5d5/json_output.0.smithi031.front.sepia.ceph.com 2023-11-29T08:57:02.694 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-11-29T08:57:02.694 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-7d4bb5d5/json_output.1.smithi031.front.sepia.ceph.com of=/dev/stdout 2023-11-29T08:57:02.721 INFO:tasks.cbt_performance:reading total cpu cycles 2023-11-29T08:57:02.722 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-7d4bb5d5 -name 'perf_stat.*' 2023-11-29T08:57:02.776 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-11-29T08:57:02.776 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-11-29T08:57:02.777 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-11-29T08:57:02.777 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-7d4bb5d5/json_output.0.smithi031.front.sepia.ceph.com of=/dev/stdout 2023-11-29T08:57:02.832 INFO:tasks.cbt_performance:reading total cpu cycles 2023-11-29T08:57:02.832 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-7d4bb5d5 -name 'perf_stat.*' 2023-11-29T08:57:02.887 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-11-29T08:57:02.887 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-11-29T08:57:02.899 INFO:tasks.cbt_performance:Data inserted successfully. 2023-11-29T08:57:02.907 INFO:tasks.cbt_performance:Data inserted successfully. 2023-11-29T08:57:02.907 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-29T08:57:02.916 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-11-29T08:57:03.039 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:57:03.032+0000 7f2efba2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:57:03.041 INFO:teuthology.orchestra.run.smithi031.stderr:2023-11-29T08:57:03.040+0000 7f2efba2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-29T08:57:03.246 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-29T08:57:03.246 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-29T08:57:03.246 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-11-29T08:57:03.246 INFO:teuthology.orchestra.run:waiting for 300 2023-11-29T08:57:07.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-11-29T08:57:09.348 INFO:tasks.ceph.osd.0:Stopped 2023-11-29T08:57:09.348 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-11-29T08:57:09.348 INFO:teuthology.orchestra.run:waiting for 300 2023-11-29T08:57:12.943 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-11-29T08:57:12.943 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-11-29T08:57:15.450 INFO:tasks.ceph.osd.1:Stopped 2023-11-29T08:57:15.451 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-11-29T08:57:15.451 INFO:teuthology.orchestra.run:waiting for 300 2023-11-29T08:57:18.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-11-29T08:57:18.145 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-11-29T08:57:21.552 INFO:tasks.ceph.osd.2:Stopped 2023-11-29T08:57:21.552 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-29T08:57:21.553 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-11-29T08:57:21.553 INFO:teuthology.orchestra.run:waiting for 300 2023-11-29T08:57:21.585 INFO:tasks.ceph.mgr.x:Stopped 2023-11-29T08:57:21.585 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-29T08:57:21.585 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-11-29T08:57:21.585 INFO:teuthology.orchestra.run:waiting for 300 2023-11-29T08:57:21.639 INFO:tasks.ceph.mon.a:Stopped 2023-11-29T08:57:21.639 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-29T08:57:21.639 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-11-29T08:57:21.735 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:57:21.735 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-11-29T08:57:21.933 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:57:21.933 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-11-29T08:57:22.052 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:57:22.052 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-11-29T08:57:22.117 INFO:tasks.ceph:Archiving mon data... 2023-11-29T08:57:22.119 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2023-11-29_08:30:41-crimson-rados-wip-matanb-crimson-snaptrimevent-lifetime-v2-distro-crimson-smithi/7470487/data/mon.a.tgz 2023-11-29T08:57:22.119 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2023-11-29T08:57:22.143 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.VJXj3RBljv 2023-11-29T08:57:22.143 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.VJXj3RBljv 2023-11-29T08:57:22.278 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.VJXj3RBljv 2023-11-29T08:57:22.379 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.VJXj3RBljv is 206KB 2023-11-29T08:57:22.445 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.VJXj3RBljv 2023-11-29T08:57:22.460 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-29T08:57:22.460 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-29T08:57:22.599 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-11-29T08:57:22.599 INFO:tasks.ceph:Archiving crash dumps... 2023-11-29T08:57:22.601 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2023-11-29_08:30:41-crimson-rados-wip-matanb-crimson-snaptrimevent-lifetime-v2-distro-crimson-smithi/7470487/remote/smithi031/crash 2023-11-29T08:57:22.602 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-29T08:57:22.669 INFO:tasks.ceph:Compressing logs... 2023-11-29T08:57:22.669 DEBUG:teuthology.orchestra.run.smithi031:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-29T08:57:22.768 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82006.log 2023-11-29T08:57:22.768 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-11-29T08:57:22.768 INFO:teuthology.orchestra.run.smithi031.stderr:gzip/var/log/ceph/ceph-client.admin.82006.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.81221.log 2023-11-29T08:57:22.768 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86759.log 2023-11-29T08:57:22.768 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86415.log 2023-11-29T08:57:22.768 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.86759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82417.log 2023-11-29T08:57:22.768 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86759.log.gz 2023-11-29T08:57:22.768 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.86415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82062.log 2023-11-29T08:57:22.768 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81221.log.gz 2023-11-29T08:57:22.768 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.79761.log 2023-11-29T08:57:22.768 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82417.log: /var/log/ceph/ceph-client.admin.82062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86319.log 2023-11-29T08:57:22.769 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.tmp-client.admin.79761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86219.log 2023-11-29T08:57:22.769 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.86319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86366.log 2023-11-29T08:57:22.769 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.86219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86383.log 2023-11-29T08:57:22.769 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.86366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82711.log 2023-11-29T08:57:22.769 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.86383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82543.log 2023-11-29T08:57:22.769 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86342.log 2023-11-29T08:57:22.769 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86414.log 2023-11-29T08:57:22.769 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.86342.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-29T08:57:22.769 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.86414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86155.log 2023-11-29T08:57:22.769 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81599.log 2023-11-29T08:57:22.769 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.86155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86760.log 2023-11-29T08:57:22.769 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81599.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-11-29T08:57:22.769 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.86760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81331.log 2023-11-29T08:57:22.769 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86760.log.gz 2023-11-29T08:57:22.769 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82585.log 2023-11-29T08:57:22.769 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.81331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81569.log 2023-11-29T08:57:22.769 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82165.log 2023-11-29T08:57:22.770 INFO:teuthology.orchestra.run.smithi031.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.79761.log.gz 2023-11-29T08:57:22.770 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86712.log 2023-11-29T08:57:22.770 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81569.log: /var/log/ceph/ceph-client.admin.82165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82456.log 2023-11-29T08:57:22.770 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.86712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82329.log 2023-11-29T08:57:22.770 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.86712.log.gz 2023-11-29T08:57:22.770 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86246.log 2023-11-29T08:57:22.770 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82331.log 2023-11-29T08:57:22.770 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.86246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82627.log 2023-11-29T08:57:22.770 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82331.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-11-29T08:57:22.770 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82330.log 2023-11-29T08:57:22.770 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82669.log 2023-11-29T08:57:22.770 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86271.log 2023-11-29T08:57:22.770 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87049.log 2023-11-29T08:57:22.770 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.86271.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-11-29T08:57:22.770 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.87049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82249.log 2023-11-29T08:57:22.770 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.87049.log.gz 2023-11-29T08:57:22.771 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86296.log 2023-11-29T08:57:22.771 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82249.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-29T08:57:22.771 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.86296.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-11-29T08:57:22.771 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82457.log 2023-11-29T08:57:22.771 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81133.log 2023-11-29T08:57:22.771 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82207.log 2023-11-29T08:57:22.771 INFO:teuthology.orchestra.run.smithi031.stderr: 87.7% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-29T08:57:22.771 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82104.log 2023-11-29T08:57:22.771 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81411.log 2023-11-29T08:57:22.771 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.82104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81371.log 2023-11-29T08:57:22.772 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.81411.log: /var/log/ceph/ceph-client.admin.81371.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81599.log.gz 2023-11-29T08:57:22.772 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86155.log.gz 2023-11-29T08:57:22.772 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.86219.log.gz 2023-11-29T08:57:22.772 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.86246.log.gz 2023-11-29T08:57:22.772 INFO:teuthology.orchestra.run.smithi031.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-11-29T08:57:22.773 INFO:teuthology.orchestra.run.smithi031.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.81133.log.gz 2023-11-29T08:57:22.774 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82006.log.gz 2023-11-29T08:57:22.774 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82062.log.gz 2023-11-29T08:57:22.774 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82104.log.gz 2023-11-29T08:57:22.776 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82165.log.gz 2023-11-29T08:57:22.776 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82207.log.gz 2023-11-29T08:57:22.776 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82249.log.gz 2023-11-29T08:57:22.776 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82329.log.gz 2023-11-29T08:57:22.776 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82331.log.gz 2023-11-29T08:57:22.776 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82330.log.gz 2023-11-29T08:57:22.776 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82417.log.gz 2023-11-29T08:57:22.776 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82456.log.gz 2023-11-29T08:57:22.776 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82457.log.gz 2023-11-29T08:57:22.776 INFO:teuthology.orchestra.run.smithi031.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82543.log.gz 2023-11-29T08:57:22.776 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82585.log.gz 2023-11-29T08:57:22.776 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82627.log.gz 2023-11-29T08:57:22.776 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82669.log.gz 2023-11-29T08:57:22.776 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82711.log.gz 2023-11-29T08:57:22.779 INFO:teuthology.orchestra.run.smithi031.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.81331.log.gz 2023-11-29T08:57:22.779 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81371.log.gz 2023-11-29T08:57:22.779 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81411.log.gz 2023-11-29T08:57:22.784 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81569.log.gz 2023-11-29T08:57:22.789 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.86271.log.gz 2023-11-29T08:57:22.789 INFO:teuthology.orchestra.run.smithi031.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.86296.log.gz 2023-11-29T08:57:22.789 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86319.log.gz 2023-11-29T08:57:22.789 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.86342.log.gz 2023-11-29T08:57:22.790 INFO:teuthology.orchestra.run.smithi031.stderr: 65.9% -- replaced with /var/log/ceph/ceph-client.admin.86366.log.gz 2023-11-29T08:57:22.790 INFO:teuthology.orchestra.run.smithi031.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.86383.log.gz 2023-11-29T08:57:22.790 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86415.log.gz 2023-11-29T08:57:22.790 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.86414.log.gz 2023-11-29T08:57:22.903 INFO:teuthology.orchestra.run.smithi031.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-11-29T08:57:22.906 INFO:teuthology.orchestra.run.smithi031.stderr: 85.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-29T08:57:23.147 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-11-29T08:57:41.009 INFO:teuthology.orchestra.run.smithi031.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-11-29T08:57:42.434 INFO:teuthology.orchestra.run.smithi031.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-11-29T08:57:42.504 INFO:teuthology.orchestra.run.smithi031.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-11-29T08:57:42.506 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-11-29T08:57:42.506 INFO:teuthology.orchestra.run.smithi031.stderr:real 0m19.782s 2023-11-29T08:57:42.506 INFO:teuthology.orchestra.run.smithi031.stderr:user 0m13.835s 2023-11-29T08:57:42.506 INFO:teuthology.orchestra.run.smithi031.stderr:sys 0m0.662s 2023-11-29T08:57:42.507 INFO:tasks.ceph:Archiving logs... 2023-11-29T08:57:42.507 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/log/ceph to /home/teuthworker/archive/matan-2023-11-29_08:30:41-crimson-rados-wip-matanb-crimson-snaptrimevent-lifetime-v2-distro-crimson-smithi/7470487/remote/smithi031/log 2023-11-29T08:57:42.508 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-29T08:57:44.504 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-29T08:57:44.514 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-29T08:57:44.514 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-29T08:57:44.585 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-29T08:57:44.595 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-29T08:57:44.595 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-29T08:57:44.612 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2023-11-29T08:57:44.679 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-29T08:57:44.680 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2023-11-29T08:57:44.680 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 12 +952us[ +952us] +/- 77ms 2023-11-29T08:57:44.680 INFO:teuthology.orchestra.run.smithi031.stdout:^* hv02.front.sepia.ceph.com 3 6 377 13 -938us[ -939us] +/- 59ms 2023-11-29T08:57:44.680 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 11 +123us[ +123us] +/- 76ms 2023-11-29T08:57:44.680 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-29T08:57:44.687 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-29T08:57:44.698 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-29T08:57:44.699 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-29T08:57:44.707 DEBUG:teuthology.orchestra.run.smithi031:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-29T08:57:44.901 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-29T08:57:44.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-29T08:57:44.938 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-29T08:57:44.947 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-29T08:57:44.959 INFO:teuthology.task.internal:Duration was 1107.094442 seconds 2023-11-29T08:57:44.959 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-29T08:57:44.968 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-29T08:57:44.968 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-29T08:57:45.147 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-29T08:57:45.416 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-29T08:57:45.416 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2023-11-29T08:57:45.416 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-29T08:57:45.436 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-29T08:57:45.437 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-29T08:57:45.510 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-29T08:57:45.510 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-29T08:57:45.659 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-29T08:57:45.670 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-29T08:57:45.670 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-29T08:57:45.715 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-29T08:57:45.725 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-29T08:57:45.789 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2023-11-29T08:57:45.810 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-29T08:57:45.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-29T08:57:45.824 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-29T08:57:45.834 INFO:teuthology.task.internal:Transferring archived files... 2023-11-29T08:57:45.835 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2023-11-29_08:30:41-crimson-rados-wip-matanb-crimson-snaptrimevent-lifetime-v2-distro-crimson-smithi/7470487/remote/smithi031 2023-11-29T08:57:45.835 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-29T08:57:45.940 INFO:teuthology.task.internal:Removing archive directory... 2023-11-29T08:57:45.941 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-29T08:57:45.974 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-29T08:57:45.986 INFO:teuthology.task.internal:Not uploading archives. 2023-11-29T08:57:45.987 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-29T08:57:45.996 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-29T08:57:45.996 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-29T08:57:46.028 INFO:teuthology.orchestra.run.smithi031.stdout: 265244 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 29 08:57 /home/ubuntu/cephtest 2023-11-29T08:57:46.030 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-29T08:57:46.039 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-29T08:57:46.077 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/2Jx1MmfVk/fio?orgId=1&var-branch_name=wip-matanb-crimson-snaptrimevent-lifetime-v2&var-job_id_selected=7470487 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_write} duration: 1107.0944418907166 flavor: crimson owner: scheduled_matan@teuthology success: true 2023-11-29T08:57:46.077 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-29T08:57:46.162 INFO:teuthology.run:pass