2024-01-18T14:03:37.030 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-18T14:03:37.030 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/matan-2024-01-18_13:55:25-crimson-rados-wip-matanb-crimson-testing-18.1-crimson-only-distro-crimson-smithi/7520293 --name matan-2024-01-18_13:55:25-crimson-rados-wip-matanb-crimson-testing-18.1-crimson-only-distro-crimson-smithi --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} --verbose --owner scheduled_matan@teuthology -- /home/teuthworker/archive/matan-2024-01-18_13:55:25-crimson-rados-wip-matanb-crimson-testing-18.1-crimson-only-distro-crimson-smithi/7520293/orig.config.yaml 2024-01-18T14:03:37.075 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-18T14:03:37.157 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2024-01-18_13:55:25-crimson-rados-wip-matanb-crimson-testing-18.1-crimson-only-distro-crimson-smithi/7520293 branch: wip-matanb-crimson-testing-18.1-crimson-only description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} email: null first_in_suite: false job_id: '7520293' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2024-01-18_13:55:25-crimson-rados-wip-matanb-crimson-testing-18.1-crimson-only-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-testing-18.1-crimson-only ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: 45e21eee52b813a77f38a534981b5dd2d1bbb6e3 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 45e21eee52b813a77f38a534981b5dd2d1bbb6e3 workunit: branch: wip-matanb-crimson-testing-18.1-crimson-only env: CRIMSON_COMPAT: '1' sha1: 45e21eee52b813a77f38a534981b5dd2d1bbb6e3 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 8851 sha1: 45e21eee52b813a77f38a534981b5dd2d1bbb6e3 sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-matanb-crimson-testing-18.1-crimson-only suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_45e21eee52b813a77f38a534981b5dd2d1bbb6e3/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 45e21eee52b813a77f38a534981b5dd2d1bbb6e3 targets: smithi049.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAeJhrp6kQZlsvIaMgSXIYTQ1r1l9fOVudiEfd9DWwGBB5EZwMT5QoU9QzRIMnXLFGCnNnO1FKzG/SGVW/EUxuc= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: librbdfio: iodepth: - 32 log_avg_msec: 100 mode: - randread norandommap: true op_size: - 4096 osd_ra: - 4096 pool_profile: rbd procs_per_volume: - 1 time: 60 vol_size: 4096 volumes_per_client: - 2 cluster: iterations: 1 osds_per_node: 3 pool_profiles: rbd: pg_size: 128 pgp_size: 128 replication: 3 user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-18_13:55:25 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-18T14:03:37.157 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_45e21eee52b813a77f38a534981b5dd2d1bbb6e3/qa; will attempt to use it 2024-01-18T14:03:37.158 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_45e21eee52b813a77f38a534981b5dd2d1bbb6e3/qa/tasks 2024-01-18T14:03:37.158 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-18T14:03:37.160 INFO:teuthology.task.internal:Checking packages... 2024-01-18T14:03:37.183 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '45e21eee52b813a77f38a534981b5dd2d1bbb6e3' 2024-01-18T14:03:37.184 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-18T14:03:37.184 INFO:teuthology.packaging:ref: None 2024-01-18T14:03:37.184 INFO:teuthology.packaging:tag: None 2024-01-18T14:03:37.184 INFO:teuthology.packaging:branch: wip-matanb-crimson-testing-18.1-crimson-only 2024-01-18T14:03:37.184 INFO:teuthology.packaging:sha1: 45e21eee52b813a77f38a534981b5dd2d1bbb6e3 2024-01-18T14:03:37.184 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-testing-18.1-crimson-only 2024-01-18T14:03:37.326 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-749.g45e21eee 2024-01-18T14:03:37.327 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-18T14:03:37.333 INFO:teuthology.task.internal:no buildpackages task found 2024-01-18T14:03:37.333 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-18T14:03:37.341 INFO:teuthology.task.internal:Saving configuration 2024-01-18T14:03:37.357 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-18T14:03:37.363 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-18T14:03:37.388 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi049.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2024-01-18_13:55:25-crimson-rados-wip-matanb-crimson-testing-18.1-crimson-only-distro-crimson-smithi/7520293', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-18 13:57:44.222753', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAeJhrp6kQZlsvIaMgSXIYTQ1r1l9fOVudiEfd9DWwGBB5EZwMT5QoU9QzRIMnXLFGCnNnO1FKzG/SGVW/EUxuc='} 2024-01-18T14:03:37.389 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-18T14:03:37.396 INFO:teuthology.task.internal:roles: ubuntu@smithi049.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-01-18T14:03:37.397 INFO:teuthology.run_tasks:Running task console_log... 2024-01-18T14:03:37.435 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcc6bf20dc0>, signals=[15]) 2024-01-18T14:03:37.436 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-18T14:03:37.442 INFO:teuthology.task.internal:Opening connections... 2024-01-18T14:03:37.442 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:03:37.444 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-18T14:03:37.549 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-18T14:03:37.555 DEBUG:teuthology.orchestra.run.smithi049:> uname -m 2024-01-18T14:03:37.617 INFO:teuthology.orchestra.run.smithi049.stdout:x86_64 2024-01-18T14:03:37.617 DEBUG:teuthology.orchestra.run.smithi049:> cat /etc/os-release 2024-01-18T14:03:37.675 INFO:teuthology.orchestra.run.smithi049.stdout:NAME="CentOS Stream" 2024-01-18T14:03:37.675 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION="9" 2024-01-18T14:03:37.675 INFO:teuthology.orchestra.run.smithi049.stdout:ID="centos" 2024-01-18T14:03:37.675 INFO:teuthology.orchestra.run.smithi049.stdout:ID_LIKE="rhel fedora" 2024-01-18T14:03:37.675 INFO:teuthology.orchestra.run.smithi049.stdout:VERSION_ID="9" 2024-01-18T14:03:37.675 INFO:teuthology.orchestra.run.smithi049.stdout:PLATFORM_ID="platform:el9" 2024-01-18T14:03:37.676 INFO:teuthology.orchestra.run.smithi049.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-18T14:03:37.676 INFO:teuthology.orchestra.run.smithi049.stdout:ANSI_COLOR="0;31" 2024-01-18T14:03:37.676 INFO:teuthology.orchestra.run.smithi049.stdout:LOGO="fedora-logo-icon" 2024-01-18T14:03:37.676 INFO:teuthology.orchestra.run.smithi049.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-18T14:03:37.676 INFO:teuthology.orchestra.run.smithi049.stdout:HOME_URL="https://centos.org/" 2024-01-18T14:03:37.676 INFO:teuthology.orchestra.run.smithi049.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-18T14:03:37.676 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-18T14:03:37.676 INFO:teuthology.orchestra.run.smithi049.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-18T14:03:37.677 INFO:teuthology.lock.ops:Updating smithi049.front.sepia.ceph.com on lock server 2024-01-18T14:03:37.696 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-18T14:03:37.705 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-18T14:03:37.710 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-18T14:03:37.710 DEBUG:teuthology.orchestra.run.smithi049:> test '!' -e /home/ubuntu/cephtest 2024-01-18T14:03:37.732 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-18T14:03:37.738 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-18T14:03:37.738 DEBUG:teuthology.orchestra.run.smithi049:> test -z $(ls -A /var/lib/ceph) 2024-01-18T14:03:37.795 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-18T14:03:37.832 INFO:teuthology.run_tasks:Running task kernel... 2024-01-18T14:03:37.846 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-18T14:03:37.846 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-18T14:03:37.847 DEBUG:teuthology.orchestra.run.smithi049:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-18T14:03:37.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-18T14:03:37.864 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2024-01-18T14:03:37.923 INFO:teuthology.orchestra.run.smithi049.stdout:5.14.0-361.el9.x86_64 2024-01-18T14:03:37.923 INFO:teuthology.task.kernel:Running kernel on smithi049: 5.14.0-361.el9.x86_64 2024-01-18T14:03:37.923 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum install -y kernel 2024-01-18T14:03:40.007 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS-9 - AppStream 14 MB/s | 6.4 MB 00:00 2024-01-18T14:03:41.524 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS-9 - BaseOS 4.7 MB/s | 2.0 MB 00:00 2024-01-18T14:03:43.306 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - BaseOS 5.6 MB/s | 8.0 MB 00:01 2024-01-18T14:03:46.326 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - AppStream 9.9 MB/s | 19 MB 00:01 2024-01-18T14:03:50.482 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - CRB 10 MB/s | 5.9 MB 00:00 2024-01-18T14:03:51.850 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 15 kB 00:00 2024-01-18T14:03:56.362 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 4.6 MB/s | 20 MB 00:04 2024-01-18T14:04:01.400 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-01-18T14:04:03.716 INFO:teuthology.orchestra.run.smithi049.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-01-18T14:04:03.716 INFO:teuthology.orchestra.run.smithi049.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-01-18T14:04:03.820 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:04:03.829 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:04:03.830 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-01-18T14:04:03.830 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:04:03.830 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-01-18T14:04:03.830 INFO:teuthology.orchestra.run.smithi049.stdout: kernel x86_64 5.14.0-408.el9 CentOS-BaseOS 6.0 M 2024-01-18T14:04:03.830 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-01-18T14:04:03.830 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-core x86_64 5.14.0-408.el9 CentOS-BaseOS 21 M 2024-01-18T14:04:03.830 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules x86_64 5.14.0-408.el9 CentOS-BaseOS 39 M 2024-01-18T14:04:03.830 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules-core x86_64 5.14.0-408.el9 CentOS-BaseOS 33 M 2024-01-18T14:04:03.830 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:04:03.830 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-01-18T14:04:03.830 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:04:03.830 INFO:teuthology.orchestra.run.smithi049.stdout:Install 4 Packages 2024-01-18T14:04:03.830 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:04:03.831 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 99 M 2024-01-18T14:04:03.831 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 122 M 2024-01-18T14:04:03.831 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-01-18T14:04:04.377 INFO:teuthology.orchestra.run.smithi049.stdout:(1/4): kernel-5.14.0-408.el9.x86_64.rpm 11 MB/s | 6.0 MB 00:00 2024-01-18T14:04:04.769 INFO:teuthology.orchestra.run.smithi049.stdout:(2/4): kernel-core-5.14.0-408.el9.x86_64.rpm 22 MB/s | 21 MB 00:00 2024-01-18T14:04:05.594 INFO:teuthology.orchestra.run.smithi049.stdout:(3/4): kernel-modules-5.14.0-408.el9.x86_64.rpm 22 MB/s | 39 MB 00:01 2024-01-18T14:04:06.153 INFO:teuthology.orchestra.run.smithi049.stdout:(4/4): kernel-modules-core-5.14.0-408.el9.x86_6 19 MB/s | 33 MB 00:01 2024-01-18T14:04:06.155 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-01-18T14:04:06.156 INFO:teuthology.orchestra.run.smithi049.stdout:Total 43 MB/s | 99 MB 00:02 2024-01-18T14:04:06.156 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-01-18T14:04:06.597 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-01-18T14:04:06.597 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-01-18T14:04:07.656 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-01-18T14:04:07.657 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-01-18T14:04:10.196 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-01-18T14:04:10.900 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-modules-core-5.14.0-408.el9.x86_64 1/4 2024-01-18T14:04:11.028 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-core-5.14.0-408.el9.x86_64 2/4 2024-01-18T14:04:13.929 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-core-5.14.0-408.el9.x86_64 2/4 2024-01-18T14:04:14.140 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-modules-5.14.0-408.el9.x86_64 3/4 2024-01-18T14:04:19.227 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-5.14.0-408.el9.x86_64 3/4 2024-01-18T14:04:19.337 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : kernel-5.14.0-408.el9.x86_64 4/4 2024-01-18T14:04:24.450 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-core-5.14.0-408.el9.x86_64 4/4 2024-01-18T14:04:55.334 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-core-5.14.0-408.el9.x86_64 4/4 2024-01-18T14:04:55.335 INFO:teuthology.orchestra.run.smithi049.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-18T14:04:55.335 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:04:55.427 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-modules-5.14.0-408.el9.x86_64 4/4 2024-01-18T14:04:55.852 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: kernel-5.14.0-408.el9.x86_64 4/4 2024-01-18T14:04:55.853 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-5.14.0-408.el9.x86_64 1/4 2024-01-18T14:04:55.853 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-core-5.14.0-408.el9.x86_64 2/4 2024-01-18T14:04:55.853 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-modules-5.14.0-408.el9.x86_64 3/4 2024-01-18T14:04:56.285 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : kernel-modules-core-5.14.0-408.el9.x86_64 4/4 2024-01-18T14:04:56.285 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:04:56.285 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-01-18T14:04:56.285 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-5.14.0-408.el9.x86_64 2024-01-18T14:04:56.285 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-core-5.14.0-408.el9.x86_64 2024-01-18T14:04:56.285 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules-5.14.0-408.el9.x86_64 2024-01-18T14:04:56.285 INFO:teuthology.orchestra.run.smithi049.stdout: kernel-modules-core-5.14.0-408.el9.x86_64 2024-01-18T14:04:56.285 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:04:56.285 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:04:56.794 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q kernel | sort -rV | head -n 1 2024-01-18T14:04:56.929 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-5.14.0-408.el9.x86_64 2024-01-18T14:04:56.930 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-408.el9.x86_64 2024-01-18T14:04:56.930 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-408.el9.x86_64 2024-01-18T14:04:56.930 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-18T14:04:56.930 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-18T14:04:56.930 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-01-18T14:04:56.930 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi049.front.sepia.ceph.com, path=None, version=distro) 2024-01-18T14:04:56.930 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q kernel | sort -rV | head -n 1 2024-01-18T14:04:57.043 INFO:teuthology.orchestra.run.smithi049.stdout:kernel-5.14.0-408.el9.x86_64 2024-01-18T14:04:57.043 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-408.el9.x86_64 2024-01-18T14:04:57.043 DEBUG:teuthology.orchestra.run.smithi049:> sudo rpm -qi grub2-tools 2024-01-18T14:04:57.090 INFO:teuthology.orchestra.run.smithi049.stdout:Name : grub2-tools 2024-01-18T14:04:57.090 INFO:teuthology.orchestra.run.smithi049.stdout:Epoch : 1 2024-01-18T14:04:57.091 INFO:teuthology.orchestra.run.smithi049.stdout:Version : 2.06 2024-01-18T14:04:57.091 INFO:teuthology.orchestra.run.smithi049.stdout:Release : 68.el9 2024-01-18T14:04:57.091 INFO:teuthology.orchestra.run.smithi049.stdout:Architecture: x86_64 2024-01-18T14:04:57.091 INFO:teuthology.orchestra.run.smithi049.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-18T14:04:57.091 INFO:teuthology.orchestra.run.smithi049.stdout:Group : Unspecified 2024-01-18T14:04:57.091 INFO:teuthology.orchestra.run.smithi049.stdout:Size : 8264254 2024-01-18T14:04:57.091 INFO:teuthology.orchestra.run.smithi049.stdout:License : GPLv3+ 2024-01-18T14:04:57.091 INFO:teuthology.orchestra.run.smithi049.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-18T14:04:57.091 INFO:teuthology.orchestra.run.smithi049.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-18T14:04:57.091 INFO:teuthology.orchestra.run.smithi049.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-18T14:04:57.091 INFO:teuthology.orchestra.run.smithi049.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-18T14:04:57.091 INFO:teuthology.orchestra.run.smithi049.stdout:Packager : builder@centos.org 2024-01-18T14:04:57.091 INFO:teuthology.orchestra.run.smithi049.stdout:Vendor : CentOS 2024-01-18T14:04:57.092 INFO:teuthology.orchestra.run.smithi049.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-18T14:04:57.092 INFO:teuthology.orchestra.run.smithi049.stdout:Summary : Support tools for GRUB. 2024-01-18T14:04:57.092 INFO:teuthology.orchestra.run.smithi049.stdout:Description : 2024-01-18T14:04:57.092 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:04:57.092 INFO:teuthology.orchestra.run.smithi049.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-18T14:04:57.092 INFO:teuthology.orchestra.run.smithi049.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-18T14:04:57.092 INFO:teuthology.orchestra.run.smithi049.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-18T14:04:57.092 INFO:teuthology.orchestra.run.smithi049.stdout:hardware devices. 2024-01-18T14:04:57.092 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:04:57.092 INFO:teuthology.orchestra.run.smithi049.stdout:This subpackage provides tools for support of all platforms. 2024-01-18T14:04:57.093 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-18T14:04:57.093 INFO:teuthology.task.kernel:Updating grub on smithi049 to boot 5.14.0-408.el9.x86_64 2024-01-18T14:04:57.093 DEBUG:teuthology.orchestra.run.smithi049:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-18T14:04:57.768 INFO:teuthology.orchestra.run.smithi049.stderr:Generating grub configuration file ... 2024-01-18T14:04:59.163 INFO:teuthology.orchestra.run.smithi049.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-18T14:04:59.191 INFO:teuthology.orchestra.run.smithi049.stderr:done 2024-01-18T14:04:59.193 DEBUG:teuthology.orchestra.run.smithi049:> mktemp 2024-01-18T14:04:59.209 INFO:teuthology.orchestra.run.smithi049.stdout:/tmp/tmp.dmMv8L14Jr 2024-01-18T14:04:59.210 DEBUG:teuthology.orchestra.run.smithi049:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.dmMv8L14Jr 2024-01-18T14:04:59.281 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0666 /tmp/tmp.dmMv8L14Jr 2024-01-18T14:04:59.420 DEBUG:teuthology.orchestra.remote:smithi049:/tmp/tmp.dmMv8L14Jr is 6KB 2024-01-18T14:04:59.469 DEBUG:teuthology.orchestra.run.smithi049:> rm -fr /tmp/tmp.dmMv8L14Jr 2024-01-18T14:04:59.483 DEBUG:teuthology.orchestra.run.smithi049:> sudo /bin/ls /boot/loader/entries || true 2024-01-18T14:04:59.562 INFO:teuthology.orchestra.run.smithi049.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-18T14:04:59.563 INFO:teuthology.orchestra.run.smithi049.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-18T14:04:59.563 INFO:teuthology.orchestra.run.smithi049.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-18T14:04:59.563 INFO:teuthology.orchestra.run.smithi049.stdout:15949e3c88704300882482eddfd7baae-5.14.0-408.el9.x86_64.conf 2024-01-18T14:04:59.564 DEBUG:teuthology.orchestra.run.smithi049:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-408.el9.x86_64 2024-01-18T14:04:59.715 DEBUG:teuthology.orchestra.run.smithi049:> sudo shutdown -r now 2024-01-18T14:05:29.740 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-01-18T14:05:29.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-01-18T14:05:29.741 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-18T14:06:04.352 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-01-18T14:06:13.360 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-01-18T14:06:13.361 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-18T14:06:16.412 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.49 2024-01-18T14:06:28.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-01-18T14:06:28.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-18T14:07:28.487 DEBUG:teuthology.orchestra.remote:timed out 2024-01-18T14:07:43.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-01-18T14:07:43.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-18T14:07:43.887 DEBUG:teuthology.orchestra.run.smithi049:> true 2024-01-18T14:07:44.291 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-01-18T14:07:44.291 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-408.el9.x86_64"... 2024-01-18T14:07:44.291 DEBUG:teuthology.orchestra.run.smithi049:> uname -r 2024-01-18T14:07:44.307 INFO:teuthology.orchestra.run.smithi049.stdout:5.14.0-408.el9.x86_64 2024-01-18T14:07:44.307 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-408.el9.x86_64 vs 5.14.0-408.el9.x86_64 2024-01-18T14:07:44.307 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-18T14:07:44.307 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-18T14:07:45.308 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-01-18T14:07:45.308 DEBUG:teuthology.orchestra.run.smithi049:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-18T14:07:45.393 INFO:teuthology.orchestra.run.smithi049.stdout:ttyS1 2024-01-18T14:07:45.429 DEBUG:teuthology.parallel:result is None 2024-01-18T14:07:45.429 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-18T14:07:45.437 INFO:teuthology.task.internal:Creating test directory... 2024-01-18T14:07:45.437 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-18T14:07:45.456 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-18T14:07:45.462 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-18T14:07:45.473 INFO:teuthology.task.internal:Creating archive directory... 2024-01-18T14:07:45.473 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-18T14:07:45.523 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-18T14:07:45.530 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-18T14:07:45.530 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-18T14:07:45.603 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-18T14:07:45.621 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-18T14:07:45.623 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-18T14:07:45.630 INFO:teuthology.task.internal:Configuring sudo... 2024-01-18T14:07:45.630 DEBUG:teuthology.orchestra.run.smithi049:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-18T14:07:45.663 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-18T14:07:45.672 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-18T14:07:45.672 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-18T14:07:45.719 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-18T14:07:45.804 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-18T14:07:45.860 DEBUG:teuthology.orchestra.run.smithi049:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-18T14:07:45.942 DEBUG:teuthology.orchestra.run.smithi049:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-18T14:07:45.991 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-01-18T14:07:45.991 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-18T14:07:46.062 DEBUG:teuthology.orchestra.run.smithi049:> sudo service rsyslog restart 2024-01-18T14:07:46.145 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-18T14:07:46.328 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-18T14:07:46.336 INFO:teuthology.task.internal:Starting timer... 2024-01-18T14:07:46.336 INFO:teuthology.run_tasks:Running task pcp... 2024-01-18T14:07:46.359 INFO:teuthology.run_tasks:Running task selinux... 2024-01-18T14:07:46.367 DEBUG:teuthology.orchestra.run.smithi049:> sudo service auditd rotate 2024-01-18T14:07:46.435 INFO:teuthology.orchestra.run.smithi049.stdout:Rotating logs: 2024-01-18T14:07:46.437 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-18T14:07:46.437 DEBUG:teuthology.orchestra.run.smithi049:> /usr/sbin/getenforce 2024-01-18T14:07:46.477 INFO:teuthology.orchestra.run.smithi049.stdout:Permissive 2024-01-18T14:07:46.478 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi049.front.sepia.ceph.com': 'permissive'} 2024-01-18T14:07:46.478 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-18T14:07:46.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-18T14:07:46.550 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-18T14:07:46.550 DEBUG:teuthology.orchestra.run.smithi049:> sudo /usr/sbin/setenforce permissive 2024-01-18T14:07:46.630 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-18T14:07:46.642 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-18T14:07:46.658 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-18T14:07:46.661 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi049.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-18T14:11:52.472 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi049.front.sepia.ceph.com')] 2024-01-18T14:11:52.473 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-01-18T14:11:52.475 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi049.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-18T14:11:52.580 DEBUG:teuthology.orchestra.run.smithi049:> true 2024-01-18T14:11:52.663 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi049.front.sepia.ceph.com' 2024-01-18T14:11:52.663 INFO:teuthology.run_tasks:Running task clock... 2024-01-18T14:11:52.676 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-18T14:11:52.676 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-18T14:11:52.676 DEBUG:teuthology.orchestra.run.smithi049:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-18T14:11:52.739 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-18T14:11:52.759 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-18T14:11:52.796 INFO:teuthology.orchestra.run.smithi049.stderr:sudo: ntpd: command not found 2024-01-18T14:11:52.814 INFO:teuthology.orchestra.run.smithi049.stdout:506 Cannot talk to daemon 2024-01-18T14:11:52.834 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-18T14:11:52.854 INFO:teuthology.orchestra.run.smithi049.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-18T14:11:52.917 INFO:teuthology.orchestra.run.smithi049.stderr:bash: line 1: ntpq: command not found 2024-01-18T14:11:52.920 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-18T14:11:52.920 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2024-01-18T14:11:52.920 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T14:11:52.920 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T14:11:52.920 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T14:11:52.920 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T14:11:52.921 INFO:teuthology.run_tasks:Running task install... 2024-01-18T14:11:52.929 DEBUG:teuthology.task.install:project ceph 2024-01-18T14:11:52.929 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '45e21eee52b813a77f38a534981b5dd2d1bbb6e3'}} 2024-01-18T14:11:52.929 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '45e21eee52b813a77f38a534981b5dd2d1bbb6e3'} 2024-01-18T14:11:52.929 INFO:teuthology.task.install:Using flavor: crimson 2024-01-18T14:11:52.935 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-18T14:11:52.935 INFO:teuthology.task.install:extra packages: [] 2024-01-18T14:11:52.936 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '45e21eee52b813a77f38a534981b5dd2d1bbb6e3', 'tag': None, 'wait_for_package': False} 2024-01-18T14:11:52.936 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=45e21eee52b813a77f38a534981b5dd2d1bbb6e3 2024-01-18T14:11:53.293 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-18.1-crimson-only/45e21eee52b813a77f38a534981b5dd2d1bbb6e3/centos/9/flavors/crimson/ 2024-01-18T14:11:53.293 INFO:teuthology.task.install.rpm:Package version is 19.0.0-749.g45e21eee 2024-01-18T14:11:53.400 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-18.1-crimson-only/45e21eee52b813a77f38a534981b5dd2d1bbb6e3/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-18.1-crimson-only/45e21eee52b813a77f38a534981b5dd2d1bbb6e3/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-testing-18.1-crimson-only/45e21eee52b813a77f38a534981b5dd2d1bbb6e3/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-18T14:11:53.400 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-01-18T14:11:53.400 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-18T14:11:53.432 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-18T14:11:53.432 DEBUG:teuthology.orchestra.run.smithi049:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/45e21eee52b813a77f38a534981b5dd2d1bbb6e3/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-18T14:11:53.504 DEBUG:teuthology.orchestra.run.smithi049:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-18T14:11:53.590 DEBUG:teuthology.orchestra.run.smithi049:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-18T14:11:53.619 INFO:teuthology.orchestra.run.smithi049.stdout:check_obsoletes = 1 2024-01-18T14:11:53.621 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum clean all 2024-01-18T14:11:53.873 INFO:teuthology.orchestra.run.smithi049.stdout:73 files removed 2024-01-18T14:11:53.899 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-radosgw 2024-01-18T14:11:54.564 INFO:teuthology.orchestra.run.smithi049.stdout:ceph packages for x86_64 253 kB/s | 83 kB 00:00 2024-01-18T14:11:55.048 INFO:teuthology.orchestra.run.smithi049.stdout:ceph noarch packages 22 kB/s | 9.5 kB 00:00 2024-01-18T14:11:55.516 INFO:teuthology.orchestra.run.smithi049.stdout:ceph source packages 4.5 kB/s | 1.9 kB 00:00 2024-01-18T14:11:56.101 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS-9 - AppStream 12 MB/s | 6.4 MB 00:00 2024-01-18T14:11:57.610 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS-9 - BaseOS 5.0 MB/s | 2.0 MB 00:00 2024-01-18T14:11:59.809 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.0 MB 00:01 2024-01-18T14:12:09.976 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - AppStream 2.1 MB/s | 19 MB 00:09 2024-01-18T14:12:14.851 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - CRB 4.7 MB/s | 5.9 MB 00:01 2024-01-18T14:12:16.811 INFO:teuthology.orchestra.run.smithi049.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 15 kB 00:00 2024-01-18T14:12:18.173 INFO:teuthology.orchestra.run.smithi049.stdout:Extra Packages for Enterprise Linux 16 MB/s | 20 MB 00:01 2024-01-18T14:12:23.321 INFO:teuthology.orchestra.run.smithi049.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-01-18T14:12:25.991 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:12:25.995 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:12:25.995 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-01-18T14:12:25.995 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:12:25.995 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-01-18T14:12:25.995 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw x86_64 2:19.0.0-749.g45e21eee.el9 ceph 8.4 M 2024-01-18T14:12:25.995 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrading: 2024-01-18T14:12:25.995 INFO:teuthology.orchestra.run.smithi049.stdout: librados2 x86_64 2:19.0.0-749.g45e21eee.el9 ceph 3.1 M 2024-01-18T14:12:25.995 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1 x86_64 2:19.0.0-749.g45e21eee.el9 ceph 3.1 M 2024-01-18T14:12:25.995 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-01-18T14:12:25.995 INFO:teuthology.orchestra.run.smithi049.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-18T14:12:25.995 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base x86_64 2:19.0.0-749.g45e21eee.el9 ceph 5.1 M 2024-01-18T14:12:25.995 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common x86_64 2:19.0.0-749.g45e21eee.el9 ceph 18 M 2024-01-18T14:12:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux x86_64 2:19.0.0-749.g45e21eee.el9 ceph 25 k 2024-01-18T14:12:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-18T14:12:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-18T14:12:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-18T14:12:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2 x86_64 2:19.0.0-749.g45e21eee.el9 ceph 752 k 2024-01-18T14:12:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-18T14:12:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-18T14:12:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-18T14:12:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1 x86_64 2:19.0.0-749.g45e21eee.el9 ceph 496 k 2024-01-18T14:12:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-18T14:12:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2 x86_64 2:19.0.0-749.g45e21eee.el9 ceph 4.5 M 2024-01-18T14:12:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-18T14:12:25.996 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-18T14:12:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-18T14:12:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse x86_64 2:19.0.0-749.g45e21eee.el9 ceph 45 k 2024-01-18T14:12:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common x86_64 2:19.0.0-749.g45e21eee.el9 ceph 130 k 2024-01-18T14:12:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs x86_64 2:19.0.0-749.g45e21eee.el9 ceph 161 k 2024-01-18T14:12:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-18T14:12:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados x86_64 2:19.0.0-749.g45e21eee.el9 ceph 320 k 2024-01-18T14:12:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd x86_64 2:19.0.0-749.g45e21eee.el9 ceph 298 k 2024-01-18T14:12:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw x86_64 2:19.0.0-749.g45e21eee.el9 ceph 99 k 2024-01-18T14:12:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-18T14:12:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-18T14:12:25.997 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:12:25.997 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-01-18T14:12:25.997 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:12:25.997 INFO:teuthology.orchestra.run.smithi049.stdout:Install 27 Packages 2024-01-18T14:12:25.997 INFO:teuthology.orchestra.run.smithi049.stdout:Upgrade 2 Packages 2024-01-18T14:12:25.998 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:12:25.998 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 53 M 2024-01-18T14:12:25.998 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-01-18T14:12:27.597 INFO:teuthology.orchestra.run.smithi049.stdout:(1/29): ceph-base-19.0.0-749.g45e21eee.el9.x86_ 3.3 MB/s | 5.1 MB 00:01 2024-01-18T14:12:27.663 INFO:teuthology.orchestra.run.smithi049.stdout:(2/29): ceph-selinux-19.0.0-749.g45e21eee.el9.x 377 kB/s | 25 kB 00:00 2024-01-18T14:12:27.889 INFO:teuthology.orchestra.run.smithi049.stdout:(3/29): libcephfs2-19.0.0-749.g45e21eee.el9.x86 3.3 MB/s | 752 kB 00:00 2024-01-18T14:12:28.106 INFO:teuthology.orchestra.run.smithi049.stdout:(4/29): ceph-radosgw-19.0.0-749.g45e21eee.el9.x 4.1 MB/s | 8.4 MB 00:02 2024-01-18T14:12:28.131 INFO:teuthology.orchestra.run.smithi049.stdout:(5/29): libradosstriper1-19.0.0-749.g45e21eee.e 2.0 MB/s | 496 kB 00:00 2024-01-18T14:12:28.189 INFO:teuthology.orchestra.run.smithi049.stdout:(6/29): python3-ceph-argparse-19.0.0-749.g45e21 772 kB/s | 45 kB 00:00 2024-01-18T14:12:28.256 INFO:teuthology.orchestra.run.smithi049.stdout:(7/29): python3-ceph-common-19.0.0-749.g45e21ee 1.9 MB/s | 130 kB 00:00 2024-01-18T14:12:28.323 INFO:teuthology.orchestra.run.smithi049.stdout:(8/29): python3-cephfs-19.0.0-749.g45e21eee.el9 2.4 MB/s | 161 kB 00:00 2024-01-18T14:12:28.407 INFO:teuthology.orchestra.run.smithi049.stdout:(9/29): python3-rados-19.0.0-749.g45e21eee.el9. 3.8 MB/s | 320 kB 00:00 2024-01-18T14:12:28.491 INFO:teuthology.orchestra.run.smithi049.stdout:(10/29): python3-rbd-19.0.0-749.g45e21eee.el9.x 3.5 MB/s | 298 kB 00:00 2024-01-18T14:12:28.566 INFO:teuthology.orchestra.run.smithi049.stdout:(11/29): librgw2-19.0.0-749.g45e21eee.el9.x86_6 9.8 MB/s | 4.5 MB 00:00 2024-01-18T14:12:28.583 INFO:teuthology.orchestra.run.smithi049.stdout:(12/29): python3-rgw-19.0.0-749.g45e21eee.el9.x 1.1 MB/s | 99 kB 00:00 2024-01-18T14:12:28.833 INFO:teuthology.orchestra.run.smithi049.stdout:(13/29): ceph-common-19.0.0-749.g45e21eee.el9.x 6.3 MB/s | 18 MB 00:02 2024-01-18T14:12:28.858 INFO:teuthology.orchestra.run.smithi049.stdout:(14/29): boost-program-options-1.75.0-8.el9.x86 362 kB/s | 106 kB 00:00 2024-01-18T14:12:28.884 INFO:teuthology.orchestra.run.smithi049.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 887 kB/s | 44 kB 00:00 2024-01-18T14:12:28.909 INFO:teuthology.orchestra.run.smithi049.stdout:(16/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 530 kB/s | 159 kB 00:00 2024-01-18T14:12:28.959 INFO:teuthology.orchestra.run.smithi049.stdout:(17/29): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 661 kB 00:00 2024-01-18T14:12:28.984 INFO:teuthology.orchestra.run.smithi049.stdout:(18/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 282 kB 00:00 2024-01-18T14:12:29.010 INFO:teuthology.orchestra.run.smithi049.stdout:(19/29): python3-prettytable-0.7.2-27.el9.noarc 412 kB/s | 41 kB 00:00 2024-01-18T14:12:29.127 INFO:teuthology.orchestra.run.smithi049.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 223 kB/s | 26 kB 00:00 2024-01-18T14:12:29.235 INFO:teuthology.orchestra.run.smithi049.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 449 kB/s | 49 kB 00:00 2024-01-18T14:12:29.353 INFO:teuthology.orchestra.run.smithi049.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 575 kB/s | 67 kB 00:00 2024-01-18T14:12:29.378 INFO:teuthology.orchestra.run.smithi049.stdout:(23/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 739 kB/s | 309 kB 00:00 2024-01-18T14:12:29.511 INFO:teuthology.orchestra.run.smithi049.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-01-18T14:12:29.803 INFO:teuthology.orchestra.run.smithi049.stdout:(25/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.8 MB/s | 838 kB 00:00 2024-01-18T14:12:29.962 INFO:teuthology.orchestra.run.smithi049.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 3.5 MB/s | 1.6 MB 00:00 2024-01-18T14:12:30.221 INFO:teuthology.orchestra.run.smithi049.stdout:(27/29): librados2-19.0.0-749.g45e21eee.el9.x86 7.5 MB/s | 3.1 MB 00:00 2024-01-18T14:12:30.279 INFO:teuthology.orchestra.run.smithi049.stdout:(28/29): librbd1-19.0.0-749.g45e21eee.el9.x86_6 9.7 MB/s | 3.1 MB 00:00 2024-01-18T14:12:30.354 INFO:teuthology.orchestra.run.smithi049.stdout:(29/29): libarrow-9.0.0-10.el9.x86_64.rpm 3.3 MB/s | 4.4 MB 00:01 2024-01-18T14:12:30.359 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-01-18T14:12:30.359 INFO:teuthology.orchestra.run.smithi049.stdout:Total 12 MB/s | 53 MB 00:04 2024-01-18T14:12:30.359 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-01-18T14:12:30.398 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-01-18T14:12:30.398 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-01-18T14:12:30.864 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-01-18T14:12:30.865 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-01-18T14:12:32.249 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-01-18T14:12:32.616 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-01-18T14:12:32.673 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librados2-2:19.0.0-749.g45e21eee.el9.x86_64 2/31 2024-01-18T14:12:32.766 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-2:19.0.0-749.g45e21eee.el9.x86_64 2/31 2024-01-18T14:12:32.858 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rados-2:19.0.0-749.g45e21eee.el9.x86_64 3/31 2024-01-18T14:12:32.883 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs2-2:19.0.0-749.g45e21eee.el9.x86_64 4/31 2024-01-18T14:12:32.974 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephfs2-2:19.0.0-749.g45e21eee.el9.x86_64 4/31 2024-01-18T14:12:33.004 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/31 2024-01-18T14:12:33.029 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/31 2024-01-18T14:12:33.067 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/31 2024-01-18T14:12:33.111 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-argparse-2:19.0.0-749.g45e21eee.el9.x 8/31 2024-01-18T14:12:33.183 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cephfs-2:19.0.0-749.g45e21eee.el9.x86_64 9/31 2024-01-18T14:12:33.212 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libradosstriper1-2:19.0.0-749.g45e21eee.el9.x86_64 10/31 2024-01-18T14:12:33.300 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libradosstriper1-2:19.0.0-749.g45e21eee.el9.x86_64 10/31 2024-01-18T14:12:33.335 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/31 2024-01-18T14:12:33.447 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-01-18T14:12:33.478 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-01-18T14:12:33.519 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-01-18T14:12:33.551 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-01-18T14:12:33.580 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-01-18T14:12:33.613 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-01-18T14:12:33.960 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-01-18T14:12:33.993 INFO:teuthology.orchestra.run.smithi049.stdout: Upgrading : librbd1-2:19.0.0-749.g45e21eee.el9.x86_64 19/31 2024-01-18T14:12:34.083 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-2:19.0.0-749.g45e21eee.el9.x86_64 19/31 2024-01-18T14:12:34.114 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rbd-2:19.0.0-749.g45e21eee.el9.x86_64 20/31 2024-01-18T14:12:34.160 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-01-18T14:12:34.614 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-01-18T14:12:34.637 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librgw2-2:19.0.0-749.g45e21eee.el9.x86_64 23/31 2024-01-18T14:12:34.708 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librgw2-2:19.0.0-749.g45e21eee.el9.x86_64 23/31 2024-01-18T14:12:34.779 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rgw-2:19.0.0-749.g45e21eee.el9.x86_64 24/31 2024-01-18T14:12:34.806 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ceph-common-2:19.0.0-749.g45e21eee.el9.x86 25/31 2024-01-18T14:12:36.931 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:19.0.0-749.g45e21eee.el9.x86_64 26/31 2024-01-18T14:12:36.969 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-common-2:19.0.0-749.g45e21eee.el9.x86_64 26/31 2024-01-18T14:12:37.538 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-common-2:19.0.0-749.g45e21eee.el9.x86_64 26/31 2024-01-18T14:12:37.581 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-base-2:19.0.0-749.g45e21eee.el9.x86_64 27/31 2024-01-18T14:12:37.675 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-base-2:19.0.0-749.g45e21eee.el9.x86_64 27/31 2024-01-18T14:12:37.675 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-18T14:12:37.675 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-18T14:12:37.675 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:12:37.707 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-selinux-2:19.0.0-749.g45e21eee.el9.x86_64 28/31 2024-01-18T14:12:47.890 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-selinux-2:19.0.0-749.g45e21eee.el9.x86_64 28/31 2024-01-18T14:12:47.890 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /sys 2024-01-18T14:12:47.890 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /proc 2024-01-18T14:12:47.890 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /mnt 2024-01-18T14:12:47.890 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /var/tmp 2024-01-18T14:12:47.890 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /home 2024-01-18T14:12:47.890 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /root 2024-01-18T14:12:47.890 INFO:teuthology.orchestra.run.smithi049.stdout:skipping the directory /tmp 2024-01-18T14:12:47.890 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:12:48.757 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-radosgw-2:19.0.0-749.g45e21eee.el9.x86_64 29/31 2024-01-18T14:12:48.783 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-749.g45e21eee.el9.x86_64 29/31 2024-01-18T14:12:48.783 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-18T14:12:48.783 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-18T14:12:48.783 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-18T14:12:48.783 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-18T14:12:48.784 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:12:48.785 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-18T14:12:48.868 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-18T14:12:48.869 INFO:teuthology.orchestra.run.smithi049.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-18T14:12:50.010 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-18T14:12:50.010 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-base-2:19.0.0-749.g45e21eee.el9.x86_64 1/31 2024-01-18T14:12:50.010 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-common-2:19.0.0-749.g45e21eee.el9.x86_64 2/31 2024-01-18T14:12:50.010 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-radosgw-2:19.0.0-749.g45e21eee.el9.x86_64 3/31 2024-01-18T14:12:50.010 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-selinux-2:19.0.0-749.g45e21eee.el9.x86_64 4/31 2024-01-18T14:12:50.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs2-2:19.0.0-749.g45e21eee.el9.x86_64 5/31 2024-01-18T14:12:50.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libradosstriper1-2:19.0.0-749.g45e21eee.el9.x86_64 6/31 2024-01-18T14:12:50.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librgw2-2:19.0.0-749.g45e21eee.el9.x86_64 7/31 2024-01-18T14:12:50.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-argparse-2:19.0.0-749.g45e21eee.el9.x 8/31 2024-01-18T14:12:50.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ceph-common-2:19.0.0-749.g45e21eee.el9.x86 9/31 2024-01-18T14:12:50.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cephfs-2:19.0.0-749.g45e21eee.el9.x86_64 10/31 2024-01-18T14:12:50.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rados-2:19.0.0-749.g45e21eee.el9.x86_64 11/31 2024-01-18T14:12:50.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rbd-2:19.0.0-749.g45e21eee.el9.x86_64 12/31 2024-01-18T14:12:50.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rgw-2:19.0.0-749.g45e21eee.el9.x86_64 13/31 2024-01-18T14:12:50.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-01-18T14:12:50.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-01-18T14:12:50.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-01-18T14:12:50.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-01-18T14:12:50.011 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-01-18T14:12:50.012 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-01-18T14:12:50.012 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-01-18T14:12:50.012 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-01-18T14:12:50.012 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-01-18T14:12:50.012 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-01-18T14:12:50.012 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-01-18T14:12:50.012 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-01-18T14:12:50.012 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-01-18T14:12:50.012 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-01-18T14:12:50.012 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-2:19.0.0-749.g45e21eee.el9.x86_64 28/31 2024-01-18T14:12:50.012 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-01-18T14:12:50.013 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-2:19.0.0-749.g45e21eee.el9.x86_64 30/31 2024-01-18T14:12:50.954 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-01-18T14:12:50.954 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:12:50.954 INFO:teuthology.orchestra.run.smithi049.stdout:Upgraded: 2024-01-18T14:12:50.954 INFO:teuthology.orchestra.run.smithi049.stdout: librados2-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:12:50.954 INFO:teuthology.orchestra.run.smithi049.stdout: librbd1-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:12:50.954 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-01-18T14:12:50.954 INFO:teuthology.orchestra.run.smithi049.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-18T14:12:50.955 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-base-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:12:50.955 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-common-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:12:50.955 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-radosgw-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:12:50.955 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-selinux-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:12:50.955 INFO:teuthology.orchestra.run.smithi049.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-18T14:12:50.955 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-18T14:12:50.955 INFO:teuthology.orchestra.run.smithi049.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-18T14:12:50.955 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs2-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:12:50.955 INFO:teuthology.orchestra.run.smithi049.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-18T14:12:50.955 INFO:teuthology.orchestra.run.smithi049.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-18T14:12:50.955 INFO:teuthology.orchestra.run.smithi049.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-18T14:12:50.955 INFO:teuthology.orchestra.run.smithi049.stdout: libradosstriper1-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:12:50.955 INFO:teuthology.orchestra.run.smithi049.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-18T14:12:50.955 INFO:teuthology.orchestra.run.smithi049.stdout: librgw2-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:12:50.955 INFO:teuthology.orchestra.run.smithi049.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-18T14:12:50.956 INFO:teuthology.orchestra.run.smithi049.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-18T14:12:50.956 INFO:teuthology.orchestra.run.smithi049.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-18T14:12:50.956 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-argparse-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:12:50.956 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ceph-common-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:12:50.956 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cephfs-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:12:50.956 INFO:teuthology.orchestra.run.smithi049.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-18T14:12:50.956 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rados-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:12:50.956 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rbd-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:12:50.956 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rgw-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:12:50.956 INFO:teuthology.orchestra.run.smithi049.stdout: re2-1:20211101-3.el9.x86_64 2024-01-18T14:12:50.956 INFO:teuthology.orchestra.run.smithi049.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-18T14:12:50.956 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:12:50.956 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:12:51.289 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-test 2024-01-18T14:12:51.889 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:00:28 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:12:52.056 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:12:52.057 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:12:52.057 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-01-18T14:12:52.057 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:12:52.057 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-01-18T14:12:52.057 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test x86_64 2:19.0.0-749.g45e21eee.el9 ceph 119 M 2024-01-18T14:12:52.057 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-01-18T14:12:52.058 INFO:teuthology.orchestra.run.smithi049.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2024-01-18T14:12:52.058 INFO:teuthology.orchestra.run.smithi049.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-18T14:12:52.058 INFO:teuthology.orchestra.run.smithi049.stdout: libasan x86_64 11.4.1-3.el9 CentOS-AppStream 417 k 2024-01-18T14:12:52.058 INFO:teuthology.orchestra.run.smithi049.stdout: libcephsqlite x86_64 2:19.0.0-749.g45e21eee.el9 ceph 158 k 2024-01-18T14:12:52.058 INFO:teuthology.orchestra.run.smithi049.stdout: libubsan x86_64 11.4.1-3.el9 CentOS-AppStream 186 k 2024-01-18T14:12:52.058 INFO:teuthology.orchestra.run.smithi049.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-18T14:12:52.058 INFO:teuthology.orchestra.run.smithi049.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-18T14:12:52.058 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-18T14:12:52.058 INFO:teuthology.orchestra.run.smithi049.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-01-18T14:12:52.058 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:12:52.058 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-01-18T14:12:52.058 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:12:52.058 INFO:teuthology.orchestra.run.smithi049.stdout:Install 10 Packages 2024-01-18T14:12:52.059 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:12:52.059 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 122 M 2024-01-18T14:12:52.059 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 1.1 G 2024-01-18T14:12:52.059 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-01-18T14:12:52.383 INFO:teuthology.orchestra.run.smithi049.stdout:(1/10): libasan-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 417 kB 00:00 2024-01-18T14:12:52.424 INFO:teuthology.orchestra.run.smithi049.stdout:(2/10): libubsan-11.4.1-3.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2024-01-18T14:12:52.476 INFO:teuthology.orchestra.run.smithi049.stdout:(3/10): socat-1.7.4.1-5.el9.x86_64.rpm 5.8 MB/s | 300 kB 00:00 2024-01-18T14:12:52.508 INFO:teuthology.orchestra.run.smithi049.stdout:(4/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-01-18T14:12:52.550 INFO:teuthology.orchestra.run.smithi049.stdout:(5/10): jq-1.6-16.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2024-01-18T14:12:52.584 INFO:teuthology.orchestra.run.smithi049.stdout:(6/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 6.3 MB/s | 217 kB 00:00 2024-01-18T14:12:52.626 INFO:teuthology.orchestra.run.smithi049.stdout:(7/10): libcephsqlite-19.0.0-749.g45e21eee.el9. 390 kB/s | 158 kB 00:00 2024-01-18T14:12:53.085 INFO:teuthology.orchestra.run.smithi049.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 266 kB/s | 122 kB 00:00 2024-01-18T14:12:53.293 INFO:teuthology.orchestra.run.smithi049.stdout:(9/10): cryptopp-8.6.0-1.el9.x86_64.rpm 1.8 MB/s | 1.3 MB 00:00 2024-01-18T14:13:02.743 INFO:teuthology.orchestra.run.smithi049.stdout:(10/10): ceph-test-19.0.0-749.g45e21eee.el9.x86 11 MB/s | 119 MB 00:10 2024-01-18T14:13:02.747 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-01-18T14:13:02.747 INFO:teuthology.orchestra.run.smithi049.stdout:Total 11 MB/s | 122 MB 00:10 2024-01-18T14:13:02.747 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-01-18T14:13:02.771 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-01-18T14:13:02.771 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-01-18T14:13:03.638 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-01-18T14:13:03.638 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-01-18T14:13:04.798 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-01-18T14:13:04.897 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2024-01-18T14:13:04.953 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2024-01-18T14:13:04.991 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/10 2024-01-18T14:13:05.017 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : jq-1.6-16.el9.x86_64 4/10 2024-01-18T14:13:05.063 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/10 2024-01-18T14:13:05.092 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/10 2024-01-18T14:13:05.120 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 7/10 2024-01-18T14:13:05.170 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libasan-11.4.1-3.el9.x86_64 8/10 2024-01-18T14:13:05.197 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephsqlite-2:19.0.0-749.g45e21eee.el9.x86_64 9/10 2024-01-18T14:13:22.724 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephsqlite-2:19.0.0-749.g45e21eee.el9.x86_64 9/10 2024-01-18T14:13:22.805 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-test-2:19.0.0-749.g45e21eee.el9.x86_64 10/10 2024-01-18T14:13:31.013 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-test-2:19.0.0-749.g45e21eee.el9.x86_64 10/10 2024-01-18T14:13:31.013 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-test-2:19.0.0-749.g45e21eee.el9.x86_64 1/10 2024-01-18T14:13:31.013 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephsqlite-2:19.0.0-749.g45e21eee.el9.x86_64 2/10 2024-01-18T14:13:31.013 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libasan-11.4.1-3.el9.x86_64 3/10 2024-01-18T14:13:31.013 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 4/10 2024-01-18T14:13:31.013 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/10 2024-01-18T14:13:31.013 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/10 2024-01-18T14:13:31.013 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : jq-1.6-16.el9.x86_64 7/10 2024-01-18T14:13:31.013 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 8/10 2024-01-18T14:13:31.014 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2024-01-18T14:13:31.470 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2024-01-18T14:13:31.470 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:13:31.470 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-01-18T14:13:31.470 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-test-2:19.0.0-749.g45e21eee.el9.x86_64 cryptopp-8.6.0-1.el9.x86_64 2024-01-18T14:13:31.470 INFO:teuthology.orchestra.run.smithi049.stdout: jq-1.6-16.el9.x86_64 libasan-11.4.1-3.el9.x86_64 2024-01-18T14:13:31.470 INFO:teuthology.orchestra.run.smithi049.stdout: libcephsqlite-2:19.0.0-749.g45e21eee.el9.x86_64 libubsan-11.4.1-3.el9.x86_64 2024-01-18T14:13:31.471 INFO:teuthology.orchestra.run.smithi049.stdout: oniguruma-6.9.6-1.el9.5.x86_64 socat-1.7.4.1-5.el9.x86_64 2024-01-18T14:13:31.471 INFO:teuthology.orchestra.run.smithi049.stdout: xmlstarlet-1.6.1-20.el9.x86_64 yaml-cpp-0.6.3-5.el9.x86_64 2024-01-18T14:13:31.471 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:13:31.471 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:13:31.802 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph 2024-01-18T14:13:32.384 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:09 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:13:32.547 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:13:32.549 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================== 2024-01-18T14:13:32.549 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-01-18T14:13:32.550 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================== 2024-01-18T14:13:32.550 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-01-18T14:13:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: ceph x86_64 2:19.0.0-749.g45e21eee.el9 ceph 6.5 k 2024-01-18T14:13:32.550 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-01-18T14:13:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds x86_64 2:19.0.0-749.g45e21eee.el9 ceph 2.0 M 2024-01-18T14:13:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr x86_64 2:19.0.0-749.g45e21eee.el9 ceph 1.5 M 2024-01-18T14:13:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core noarch 2:19.0.0-749.g45e21eee.el9 ceph-noarch 244 k 2024-01-18T14:13:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon x86_64 2:19.0.0-749.g45e21eee.el9 ceph 4.9 M 2024-01-18T14:13:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd x86_64 2:19.0.0-749.g45e21eee.el9 ceph 117 M 2024-01-18T14:13:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-01-18T14:13:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-18T14:13:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-18T14:13:32.550 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-18T14:13:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-18T14:13:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-18T14:13:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-18T14:13:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-18T14:13:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-18T14:13:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-18T14:13:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-18T14:13:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-18T14:13:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-18T14:13:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-18T14:13:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-18T14:13:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-18T14:13:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-18T14:13:32.551 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-18T14:13:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-18T14:13:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-18T14:13:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-18T14:13:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-18T14:13:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-18T14:13:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-18T14:13:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-18T14:13:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-18T14:13:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-18T14:13:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-18T14:13:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-18T14:13:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-18T14:13:32.552 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:13:32.552 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-01-18T14:13:32.553 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================== 2024-01-18T14:13:32.553 INFO:teuthology.orchestra.run.smithi049.stdout:Install 36 Packages 2024-01-18T14:13:32.553 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:13:32.553 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 130 M 2024-01-18T14:13:32.553 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 1.4 G 2024-01-18T14:13:32.553 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-01-18T14:13:33.103 INFO:teuthology.orchestra.run.smithi049.stdout:(1/36): ceph-19.0.0-749.g45e21eee.el9.x86_64.rp 31 kB/s | 6.5 kB 00:00 2024-01-18T14:13:33.411 INFO:teuthology.orchestra.run.smithi049.stdout:(2/36): ceph-mds-19.0.0-749.g45e21eee.el9.x86_6 3.9 MB/s | 2.0 MB 00:00 2024-01-18T14:13:33.662 INFO:teuthology.orchestra.run.smithi049.stdout:(3/36): ceph-mgr-19.0.0-749.g45e21eee.el9.x86_6 1.9 MB/s | 1.5 MB 00:00 2024-01-18T14:13:33.762 INFO:teuthology.orchestra.run.smithi049.stdout:(4/36): ceph-mgr-modules-core-19.0.0-749.g45e21 2.4 MB/s | 244 kB 00:00 2024-01-18T14:13:33.987 INFO:teuthology.orchestra.run.smithi049.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 830 kB/s | 187 kB 00:00 2024-01-18T14:13:34.046 INFO:teuthology.orchestra.run.smithi049.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-01-18T14:13:34.088 INFO:teuthology.orchestra.run.smithi049.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 760 kB/s | 32 kB 00:00 2024-01-18T14:13:34.129 INFO:teuthology.orchestra.run.smithi049.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-01-18T14:13:34.180 INFO:teuthology.orchestra.run.smithi049.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 771 kB/s | 39 kB 00:00 2024-01-18T14:13:34.256 INFO:teuthology.orchestra.run.smithi049.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-01-18T14:13:34.365 INFO:teuthology.orchestra.run.smithi049.stdout:(11/36): ceph-mon-19.0.0-749.g45e21eee.el9.x86_ 3.8 MB/s | 4.9 MB 00:01 2024-01-18T14:13:34.448 INFO:teuthology.orchestra.run.smithi049.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 6.0 MB/s | 1.1 MB 00:00 2024-01-18T14:13:34.473 INFO:teuthology.orchestra.run.smithi049.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 956 kB/s | 103 kB 00:00 2024-01-18T14:13:34.507 INFO:teuthology.orchestra.run.smithi049.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.1 MB/s | 124 kB 00:00 2024-01-18T14:13:34.532 INFO:teuthology.orchestra.run.smithi049.stdout:(15/36): python3-requests-2.25.1-7.el9.noarch.r 1.9 MB/s | 113 kB 00:00 2024-01-18T14:13:34.566 INFO:teuthology.orchestra.run.smithi049.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-01-18T14:13:34.666 INFO:teuthology.orchestra.run.smithi049.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 167 kB/s | 22 kB 00:00 2024-01-18T14:13:34.900 INFO:teuthology.orchestra.run.smithi049.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 453 kB/s | 151 kB 00:00 2024-01-18T14:13:34.917 INFO:teuthology.orchestra.run.smithi049.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 173 kB/s | 43 kB 00:00 2024-01-18T14:13:35.025 INFO:teuthology.orchestra.run.smithi049.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-01-18T14:13:35.100 INFO:teuthology.orchestra.run.smithi049.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 142 kB/s | 11 kB 00:00 2024-01-18T14:13:35.176 INFO:teuthology.orchestra.run.smithi049.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-01-18T14:13:35.193 INFO:teuthology.orchestra.run.smithi049.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-01-18T14:13:35.243 INFO:teuthology.orchestra.run.smithi049.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 346 kB/s | 23 kB 00:00 2024-01-18T14:13:35.259 INFO:teuthology.orchestra.run.smithi049.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 291 kB/s | 19 kB 00:00 2024-01-18T14:13:35.310 INFO:teuthology.orchestra.run.smithi049.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 294 kB/s | 20 kB 00:00 2024-01-18T14:13:35.335 INFO:teuthology.orchestra.run.smithi049.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-01-18T14:13:35.402 INFO:teuthology.orchestra.run.smithi049.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 859 kB/s | 79 kB 00:00 2024-01-18T14:13:35.436 INFO:teuthology.orchestra.run.smithi049.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-01-18T14:13:35.478 INFO:teuthology.orchestra.run.smithi049.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 218 kB/s | 16 kB 00:00 2024-01-18T14:13:35.520 INFO:teuthology.orchestra.run.smithi049.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2024-01-18T14:13:35.553 INFO:teuthology.orchestra.run.smithi049.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 474 kB/s | 36 kB 00:00 2024-01-18T14:13:35.620 INFO:teuthology.orchestra.run.smithi049.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-01-18T14:13:35.679 INFO:teuthology.orchestra.run.smithi049.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.3 MB/s | 427 kB 00:00 2024-01-18T14:13:35.712 INFO:teuthology.orchestra.run.smithi049.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-01-18T14:13:38.863 INFO:teuthology.orchestra.run.smithi049.stdout:(36/36): ceph-osd-19.0.0-749.g45e21eee.el9.x86_ 21 MB/s | 117 MB 00:05 2024-01-18T14:13:38.866 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-01-18T14:13:38.867 INFO:teuthology.orchestra.run.smithi049.stdout:Total 21 MB/s | 130 MB 00:06 2024-01-18T14:13:38.889 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-01-18T14:13:38.936 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-01-18T14:13:38.936 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-01-18T14:13:39.961 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-01-18T14:13:39.961 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-01-18T14:13:41.903 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-01-18T14:13:41.974 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-01-18T14:13:42.033 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-01-18T14:13:42.076 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-01-18T14:13:42.150 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-01-18T14:13:42.200 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-01-18T14:13:42.233 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-01-18T14:13:42.329 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-01-18T14:13:42.409 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-01-18T14:13:42.456 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-01-18T14:13:42.531 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-01-18T14:13:42.582 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-01-18T14:13:42.619 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2024-01-18T14:13:42.678 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-01-18T14:13:42.741 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-01-18T14:13:42.938 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-01-18T14:13:42.982 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-01-18T14:13:43.020 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-01-18T14:13:43.055 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-01-18T14:13:43.084 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-01-18T14:13:43.164 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-01-18T14:13:43.203 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-18T14:13:43.236 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-01-18T14:13:43.277 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-01-18T14:13:43.320 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-01-18T14:13:43.363 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-01-18T14:13:43.457 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-01-18T14:13:43.502 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-01-18T14:13:43.587 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-01-18T14:13:43.729 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-01-18T14:13:43.852 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-18T14:13:44.028 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-749.g45e21eee.el9.n 31/36 2024-01-18T14:13:44.057 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-2:19.0.0-749.g45e21eee.el9.x86_64 32/36 2024-01-18T14:13:44.083 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-2:19.0.0-749.g45e21eee.el9.x86_64 32/36 2024-01-18T14:13:44.083 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-18T14:13:44.083 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-18T14:13:44.083 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-18T14:13:44.083 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-18T14:13:44.084 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:03.530 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-osd-2:19.0.0-749.g45e21eee.el9.x86_64 33/36 2024-01-18T14:14:03.557 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-osd-2:19.0.0-749.g45e21eee.el9.x86_64 33/36 2024-01-18T14:14:03.557 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-18T14:14:03.557 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-18T14:14:03.557 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-18T14:14:03.557 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-18T14:14:03.558 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:04.039 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mon-2:19.0.0-749.g45e21eee.el9.x86_64 34/36 2024-01-18T14:14:04.063 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mon-2:19.0.0-749.g45e21eee.el9.x86_64 34/36 2024-01-18T14:14:04.064 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-18T14:14:04.064 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-18T14:14:04.064 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-18T14:14:04.064 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-18T14:14:04.064 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:04.273 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mds-2:19.0.0-749.g45e21eee.el9.x86_64 35/36 2024-01-18T14:14:04.297 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mds-2:19.0.0-749.g45e21eee.el9.x86_64 35/36 2024-01-18T14:14:04.297 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-18T14:14:04.297 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-18T14:14:04.297 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-18T14:14:04.298 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-18T14:14:04.298 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:04.366 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-2:19.0.0-749.g45e21eee.el9.x86_64 36/36 2024-01-18T14:14:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-2:19.0.0-749.g45e21eee.el9.x86_64 36/36 2024-01-18T14:14:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-2:19.0.0-749.g45e21eee.el9.x86_64 1/36 2024-01-18T14:14:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mds-2:19.0.0-749.g45e21eee.el9.x86_64 2/36 2024-01-18T14:14:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-2:19.0.0-749.g45e21eee.el9.x86_64 3/36 2024-01-18T14:14:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mon-2:19.0.0-749.g45e21eee.el9.x86_64 4/36 2024-01-18T14:14:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-osd-2:19.0.0-749.g45e21eee.el9.x86_64 5/36 2024-01-18T14:14:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-749.g45e21eee.el9.n 6/36 2024-01-18T14:14:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-01-18T14:14:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-01-18T14:14:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-01-18T14:14:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-01-18T14:14:14.242 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-01-18T14:14:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-01-18T14:14:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-01-18T14:14:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-01-18T14:14:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-01-18T14:14:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2024-01-18T14:14:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-01-18T14:14:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-01-18T14:14:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-01-18T14:14:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-01-18T14:14:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-18T14:14:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-01-18T14:14:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-01-18T14:14:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-01-18T14:14:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-01-18T14:14:14.243 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-01-18T14:14:14.244 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-01-18T14:14:14.245 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-01-18T14:14:14.245 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-01-18T14:14:14.245 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-18T14:14:14.245 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-01-18T14:14:14.245 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-01-18T14:14:14.245 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-01-18T14:14:14.245 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-01-18T14:14:14.245 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-01-18T14:14:15.111 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-01-18T14:14:15.111 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:15.111 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-01-18T14:14:15.111 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:14:15.111 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mds-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:14:15.111 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:14:15.111 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-modules-core-2:19.0.0-749.g45e21eee.el9.noarch 2024-01-18T14:14:15.112 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mon-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:14:15.112 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-osd-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:14:15.112 INFO:teuthology.orchestra.run.smithi049.stdout: lua-5.4.4-4.el9.x86_64 2024-01-18T14:14:15.112 INFO:teuthology.orchestra.run.smithi049.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-18T14:14:15.112 INFO:teuthology.orchestra.run.smithi049.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-18T14:14:15.112 INFO:teuthology.orchestra.run.smithi049.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-18T14:14:15.112 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-18T14:14:15.112 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-18T14:14:15.112 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-18T14:14:15.112 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-18T14:14:15.112 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-18T14:14:15.112 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-18T14:14:15.112 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-18T14:14:15.113 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-18T14:14:15.113 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-18T14:14:15.113 INFO:teuthology.orchestra.run.smithi049.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-18T14:14:15.113 INFO:teuthology.orchestra.run.smithi049.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-18T14:14:15.113 INFO:teuthology.orchestra.run.smithi049.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-18T14:14:15.113 INFO:teuthology.orchestra.run.smithi049.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-18T14:14:15.113 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-18T14:14:15.113 INFO:teuthology.orchestra.run.smithi049.stdout: python3-ply-3.11-14.el9.noarch 2024-01-18T14:14:15.113 INFO:teuthology.orchestra.run.smithi049.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-18T14:14:15.113 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-18T14:14:15.113 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-18T14:14:15.113 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-18T14:14:15.113 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-18T14:14:15.113 INFO:teuthology.orchestra.run.smithi049.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-18T14:14:15.113 INFO:teuthology.orchestra.run.smithi049.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-18T14:14:15.114 INFO:teuthology.orchestra.run.smithi049.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-18T14:14:15.114 INFO:teuthology.orchestra.run.smithi049.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-18T14:14:15.114 INFO:teuthology.orchestra.run.smithi049.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-18T14:14:15.114 INFO:teuthology.orchestra.run.smithi049.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-18T14:14:15.114 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:15.114 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:14:15.363 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-base 2024-01-18T14:14:15.952 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:52 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:14:16.056 INFO:teuthology.orchestra.run.smithi049.stdout:Package ceph-base-2:19.0.0-749.g45e21eee.el9.x86_64 is already installed. 2024-01-18T14:14:16.109 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:14:16.110 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-01-18T14:14:16.110 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:14:16.168 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install cephadm 2024-01-18T14:14:16.752 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:53 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:14:16.908 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:14:16.909 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:14:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-01-18T14:14:16.909 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:14:16.909 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-01-18T14:14:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm noarch 2:19.0.0-749.g45e21eee.el9 ceph-noarch 742 k 2024-01-18T14:14:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:16.909 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-01-18T14:14:16.909 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:14:16.909 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-01-18T14:14:16.909 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:16.909 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 742 k 2024-01-18T14:14:16.909 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 748 k 2024-01-18T14:14:16.910 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-01-18T14:14:17.287 INFO:teuthology.orchestra.run.smithi049.stdout:cephadm-19.0.0-749.g45e21eee.el9.noarch.rpm 1.9 MB/s | 742 kB 00:00 2024-01-18T14:14:17.288 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-01-18T14:14:17.288 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.9 MB/s | 742 kB 00:00 2024-01-18T14:14:17.288 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-01-18T14:14:17.294 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-01-18T14:14:17.294 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-01-18T14:14:17.300 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-01-18T14:14:17.300 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-01-18T14:14:17.372 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-01-18T14:14:17.758 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephadm-2:19.0.0-749.g45e21eee.el9.noarch 1/1 2024-01-18T14:14:17.861 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : cephadm-2:19.0.0-749.g45e21eee.el9.noarch 1/1 2024-01-18T14:14:18.443 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: cephadm-2:19.0.0-749.g45e21eee.el9.noarch 1/1 2024-01-18T14:14:18.749 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : cephadm-2:19.0.0-749.g45e21eee.el9.noarch 1/1 2024-01-18T14:14:18.749 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:18.749 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-01-18T14:14:18.749 INFO:teuthology.orchestra.run.smithi049.stdout: cephadm-2:19.0.0-749.g45e21eee.el9.noarch 2024-01-18T14:14:18.749 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:18.749 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:14:18.953 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-immutable-object-cache 2024-01-18T14:14:19.543 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:56 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:14:19.699 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:14:19.700 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:14:19.700 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-01-18T14:14:19.700 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:14:19.700 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-01-18T14:14:19.700 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-749.g45e21eee.el9 ceph 147 k 2024-01-18T14:14:19.700 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:19.700 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-01-18T14:14:19.700 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:14:19.700 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-01-18T14:14:19.700 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:19.701 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 147 k 2024-01-18T14:14:19.701 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 481 k 2024-01-18T14:14:19.701 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-01-18T14:14:19.959 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-immutable-object-cache-19.0.0-749.g45e21ee 569 kB/s | 147 kB 00:00 2024-01-18T14:14:19.960 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-01-18T14:14:19.960 INFO:teuthology.orchestra.run.smithi049.stdout:Total 566 kB/s | 147 kB 00:00 2024-01-18T14:14:19.960 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-01-18T14:14:19.968 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-01-18T14:14:19.968 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-01-18T14:14:20.024 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-01-18T14:14:20.024 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-01-18T14:14:20.289 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-01-18T14:14:20.335 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-749.g45e21eee.e 1/1 2024-01-18T14:14:20.361 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-749.g45e21eee.e 1/1 2024-01-18T14:14:20.361 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-18T14:14:20.362 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-18T14:14:20.362 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:21.358 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-749.g45e21eee.e 1/1 2024-01-18T14:14:21.358 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:21.358 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-01-18T14:14:21.358 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-immutable-object-cache-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:14:21.358 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:21.359 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:14:21.602 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr 2024-01-18T14:14:22.188 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:59 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:14:22.292 INFO:teuthology.orchestra.run.smithi049.stdout:Package ceph-mgr-2:19.0.0-749.g45e21eee.el9.x86_64 is already installed. 2024-01-18T14:14:22.346 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:14:22.348 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-01-18T14:14:22.348 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:14:22.409 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-dashboard 2024-01-18T14:14:22.998 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:01:59 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:14:23.156 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:14:23.156 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:14:23.157 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-01-18T14:14:23.157 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:14:23.157 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-01-18T14:14:23.157 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-dashboard noarch 2:19.0.0-749.g45e21eee.el9 ceph-noarch 1.7 M 2024-01-18T14:14:23.157 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-01-18T14:14:23.157 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-grafana-dashboards noarch 2:19.0.0-749.g45e21eee.el9 ceph-noarch 28 k 2024-01-18T14:14:23.157 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-prometheus-alerts noarch 2:19.0.0-749.g45e21eee.el9 ceph-noarch 15 k 2024-01-18T14:14:23.157 INFO:teuthology.orchestra.run.smithi049.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-18T14:14:23.157 INFO:teuthology.orchestra.run.smithi049.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-18T14:14:23.157 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:23.157 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-01-18T14:14:23.157 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:14:23.157 INFO:teuthology.orchestra.run.smithi049.stdout:Install 5 Packages 2024-01-18T14:14:23.157 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:23.158 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 2.0 M 2024-01-18T14:14:23.158 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 8.5 M 2024-01-18T14:14:23.158 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-01-18T14:14:23.515 INFO:teuthology.orchestra.run.smithi049.stdout:(1/5): ceph-prometheus-alerts-19.0.0-749.g45e21 76 kB/s | 15 kB 00:00 2024-01-18T14:14:23.548 INFO:teuthology.orchestra.run.smithi049.stdout:(2/5): ceph-grafana-dashboards-19.0.0-749.g45e2 122 kB/s | 28 kB 00:00 2024-01-18T14:14:23.874 INFO:teuthology.orchestra.run.smithi049.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 86 kB/s | 31 kB 00:00 2024-01-18T14:14:24.107 INFO:teuthology.orchestra.run.smithi049.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 338 kB/s | 188 kB 00:00 2024-01-18T14:14:24.366 INFO:teuthology.orchestra.run.smithi049.stdout:(5/5): ceph-mgr-dashboard-19.0.0-749.g45e21eee. 1.7 MB/s | 1.7 MB 00:01 2024-01-18T14:14:24.369 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-01-18T14:14:24.370 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-01-18T14:14:24.370 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-01-18T14:14:24.377 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-01-18T14:14:24.377 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-01-18T14:14:24.447 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-01-18T14:14:24.448 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-01-18T14:14:24.694 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-01-18T14:14:24.781 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-01-18T14:14:24.812 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-01-18T14:14:24.842 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-749.g45e21eee.el9.no 3/5 2024-01-18T14:14:25.080 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-749.g45e21eee.el9.n 4/5 2024-01-18T14:14:25.119 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-749.g45e21eee.el9.noarch 5/5 2024-01-18T14:14:25.696 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-749.g45e21eee.el9.noarch 5/5 2024-01-18T14:14:25.696 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-749.g45e21eee.el9.n 1/5 2024-01-18T14:14:25.696 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-749.g45e21eee.el9.noarch 2/5 2024-01-18T14:14:25.697 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-749.g45e21eee.el9.no 3/5 2024-01-18T14:14:25.697 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-01-18T14:14:25.998 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-01-18T14:14:25.998 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:25.998 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-01-18T14:14:25.998 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-grafana-dashboards-2:19.0.0-749.g45e21eee.el9.noarch 2024-01-18T14:14:25.999 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-dashboard-2:19.0.0-749.g45e21eee.el9.noarch 2024-01-18T14:14:25.999 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-prometheus-alerts-2:19.0.0-749.g45e21eee.el9.noarch 2024-01-18T14:14:25.999 INFO:teuthology.orchestra.run.smithi049.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-18T14:14:25.999 INFO:teuthology.orchestra.run.smithi049.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-18T14:14:25.999 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:25.999 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:14:26.232 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-18T14:14:26.818 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:03 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:14:26.978 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:14:26.979 INFO:teuthology.orchestra.run.smithi049.stdout:=========================================================================================== 2024-01-18T14:14:26.979 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-01-18T14:14:26.979 INFO:teuthology.orchestra.run.smithi049.stdout:=========================================================================================== 2024-01-18T14:14:26.979 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-01-18T14:14:26.979 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-749.g45e21eee.el9 ceph-noarch 7.4 M 2024-01-18T14:14:26.979 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-01-18T14:14:26.979 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-18T14:14:26.979 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-18T14:14:26.979 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-18T14:14:26.980 INFO:teuthology.orchestra.run.smithi049.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-18T14:14:26.980 INFO:teuthology.orchestra.run.smithi049.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-18T14:14:26.980 INFO:teuthology.orchestra.run.smithi049.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-18T14:14:26.980 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-18T14:14:26.980 INFO:teuthology.orchestra.run.smithi049.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-18T14:14:26.980 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-18T14:14:26.980 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-18T14:14:26.980 INFO:teuthology.orchestra.run.smithi049.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-18T14:14:26.980 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:26.980 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-01-18T14:14:26.980 INFO:teuthology.orchestra.run.smithi049.stdout:=========================================================================================== 2024-01-18T14:14:26.980 INFO:teuthology.orchestra.run.smithi049.stdout:Install 12 Packages 2024-01-18T14:14:26.980 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:26.981 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 37 M 2024-01-18T14:14:26.981 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 209 M 2024-01-18T14:14:26.981 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-01-18T14:14:27.264 INFO:teuthology.orchestra.run.smithi049.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 201 kB/s | 32 kB 00:00 2024-01-18T14:14:27.297 INFO:teuthology.orchestra.run.smithi049.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 541 kB/s | 17 kB 00:00 2024-01-18T14:14:27.331 INFO:teuthology.orchestra.run.smithi049.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-01-18T14:14:27.390 INFO:teuthology.orchestra.run.smithi049.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-01-18T14:14:27.556 INFO:teuthology.orchestra.run.smithi049.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-01-18T14:14:27.623 INFO:teuthology.orchestra.run.smithi049.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-01-18T14:14:27.649 INFO:teuthology.orchestra.run.smithi049.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-01-18T14:14:27.774 INFO:teuthology.orchestra.run.smithi049.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 6.4 MB/s | 802 kB 00:00 2024-01-18T14:14:27.957 INFO:teuthology.orchestra.run.smithi049.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-01-18T14:14:27.983 INFO:teuthology.orchestra.run.smithi049.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 941 kB/s | 194 kB 00:00 2024-01-18T14:14:28.166 INFO:teuthology.orchestra.run.smithi049.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-01-18T14:14:28.425 INFO:teuthology.orchestra.run.smithi049.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-7 5.6 MB/s | 7.4 MB 00:01 2024-01-18T14:14:28.427 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-01-18T14:14:28.428 INFO:teuthology.orchestra.run.smithi049.stdout:Total 26 MB/s | 37 MB 00:01 2024-01-18T14:14:28.442 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-01-18T14:14:28.469 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-01-18T14:14:28.469 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-01-18T14:14:28.869 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-01-18T14:14:28.870 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-01-18T14:14:29.603 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-01-18T14:14:29.673 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-01-18T14:14:29.712 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-01-18T14:14:29.811 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-01-18T14:14:29.849 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-01-18T14:14:30.039 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-18T14:14:30.064 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-18T14:14:30.152 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-01-18T14:14:30.856 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-01-18T14:14:30.982 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-01-18T14:14:32.296 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-01-18T14:14:33.384 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-01-18T14:14:33.414 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-749.g45e21e 12/12 2024-01-18T14:14:35.971 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-749.g45e21e 12/12 2024-01-18T14:14:35.971 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-749.g45e21e 1/12 2024-01-18T14:14:35.971 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-01-18T14:14:35.971 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-01-18T14:14:35.971 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-01-18T14:14:35.971 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-18T14:14:35.971 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-18T14:14:35.971 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-01-18T14:14:35.971 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-01-18T14:14:35.971 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-01-18T14:14:35.971 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-01-18T14:14:35.971 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-01-18T14:14:36.471 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-01-18T14:14:36.471 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:36.471 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-01-18T14:14:36.471 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-diskprediction-local-2:19.0.0-749.g45e21eee.el9.noarch 2024-01-18T14:14:36.471 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-18T14:14:36.471 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-18T14:14:36.471 INFO:teuthology.orchestra.run.smithi049.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-18T14:14:36.471 INFO:teuthology.orchestra.run.smithi049.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-18T14:14:36.471 INFO:teuthology.orchestra.run.smithi049.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-18T14:14:36.471 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-18T14:14:36.472 INFO:teuthology.orchestra.run.smithi049.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-18T14:14:36.472 INFO:teuthology.orchestra.run.smithi049.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-18T14:14:36.472 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-18T14:14:36.472 INFO:teuthology.orchestra.run.smithi049.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-18T14:14:36.472 INFO:teuthology.orchestra.run.smithi049.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-18T14:14:36.472 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:36.472 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:14:36.850 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-rook 2024-01-18T14:14:37.445 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:14 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:14:37.614 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:14:37.615 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================= 2024-01-18T14:14:37.615 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-01-18T14:14:37.615 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================= 2024-01-18T14:14:37.615 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-01-18T14:14:37.615 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-rook noarch 2:19.0.0-749.g45e21eee.el9 ceph-noarch 51 k 2024-01-18T14:14:37.615 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-01-18T14:14:37.615 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-18T14:14:37.615 INFO:teuthology.orchestra.run.smithi049.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-18T14:14:37.616 INFO:teuthology.orchestra.run.smithi049.stdout: python3-google-auth noarch 1:2.26.0-1.el9 epel 202 k 2024-01-18T14:14:37.616 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-18T14:14:37.616 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-18T14:14:37.616 INFO:teuthology.orchestra.run.smithi049.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-18T14:14:37.616 INFO:teuthology.orchestra.run.smithi049.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-18T14:14:37.616 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-18T14:14:37.616 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-18T14:14:37.616 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-18T14:14:37.616 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-18T14:14:37.616 INFO:teuthology.orchestra.run.smithi049.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-18T14:14:37.616 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:37.616 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-01-18T14:14:37.616 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================= 2024-01-18T14:14:37.616 INFO:teuthology.orchestra.run.smithi049.stdout:Install 13 Packages 2024-01-18T14:14:37.617 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:37.617 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 2.0 M 2024-01-18T14:14:37.617 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 26 M 2024-01-18T14:14:37.617 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-01-18T14:14:38.114 INFO:teuthology.orchestra.run.smithi049.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 138 kB/s | 18 kB 00:00 2024-01-18T14:14:38.147 INFO:teuthology.orchestra.run.smithi049.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 148 kB/s | 24 kB 00:00 2024-01-18T14:14:38.198 INFO:teuthology.orchestra.run.smithi049.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-01-18T14:14:38.223 INFO:teuthology.orchestra.run.smithi049.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-01-18T14:14:38.257 INFO:teuthology.orchestra.run.smithi049.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-01-18T14:14:38.282 INFO:teuthology.orchestra.run.smithi049.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 728 kB/s | 43 kB 00:00 2024-01-18T14:14:38.307 INFO:teuthology.orchestra.run.smithi049.stdout:(7/13): ceph-mgr-rook-19.0.0-749.g45e21eee.el9. 156 kB/s | 51 kB 00:00 2024-01-18T14:14:38.374 INFO:teuthology.orchestra.run.smithi049.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 163 kB/s | 15 kB 00:00 2024-01-18T14:14:38.400 INFO:teuthology.orchestra.run.smithi049.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 226 kB/s | 32 kB 00:00 2024-01-18T14:14:38.492 INFO:teuthology.orchestra.run.smithi049.stdout:(10/13): python3-google-auth-2.26.0-1.el9.noarc 1.1 MB/s | 202 kB 00:00 2024-01-18T14:14:38.517 INFO:teuthology.orchestra.run.smithi049.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 506 kB/s | 59 kB 00:00 2024-01-18T14:14:38.567 INFO:teuthology.orchestra.run.smithi049.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2024-01-18T14:14:38.676 INFO:teuthology.orchestra.run.smithi049.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.4 MB/s | 1.0 MB 00:00 2024-01-18T14:14:38.677 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-01-18T14:14:38.678 INFO:teuthology.orchestra.run.smithi049.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2024-01-18T14:14:38.678 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-01-18T14:14:38.692 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-01-18T14:14:38.692 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-01-18T14:14:38.797 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-01-18T14:14:38.797 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-01-18T14:14:39.332 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-01-18T14:14:39.513 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-01-18T14:14:39.572 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-01-18T14:14:39.627 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-01-18T14:14:39.670 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-01-18T14:14:39.715 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-01-18T14:14:39.813 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-01-18T14:14:39.922 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-google-auth-1:2.26.0-1.el9.noarch 7/13 2024-01-18T14:14:39.973 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-01-18T14:14:40.807 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-01-18T14:14:40.851 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-01-18T14:14:40.885 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-01-18T14:14:40.935 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-01-18T14:14:40.963 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-rook-2:19.0.0-749.g45e21eee.el9.noarch 13/13 2024-01-18T14:14:41.769 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-749.g45e21eee.el9.noarch 13/13 2024-01-18T14:14:41.769 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-rook-2:19.0.0-749.g45e21eee.el9.noarch 1/13 2024-01-18T14:14:41.769 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-01-18T14:14:41.769 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-01-18T14:14:41.770 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-01-18T14:14:41.770 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-01-18T14:14:41.770 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-01-18T14:14:41.770 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-01-18T14:14:41.770 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-01-18T14:14:41.771 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-01-18T14:14:41.771 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-google-auth-1:2.26.0-1.el9.noarch 10/13 2024-01-18T14:14:41.771 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-01-18T14:14:41.771 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-01-18T14:14:42.152 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-01-18T14:14:42.152 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:42.153 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-01-18T14:14:42.153 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-rook-2:19.0.0-749.g45e21eee.el9.noarch 2024-01-18T14:14:42.153 INFO:teuthology.orchestra.run.smithi049.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-18T14:14:42.153 INFO:teuthology.orchestra.run.smithi049.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-18T14:14:42.153 INFO:teuthology.orchestra.run.smithi049.stdout: python3-google-auth-1:2.26.0-1.el9.noarch 2024-01-18T14:14:42.154 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-18T14:14:42.154 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-18T14:14:42.154 INFO:teuthology.orchestra.run.smithi049.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-18T14:14:42.154 INFO:teuthology.orchestra.run.smithi049.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-18T14:14:42.154 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-18T14:14:42.154 INFO:teuthology.orchestra.run.smithi049.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-18T14:14:42.155 INFO:teuthology.orchestra.run.smithi049.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-18T14:14:42.155 INFO:teuthology.orchestra.run.smithi049.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-18T14:14:42.155 INFO:teuthology.orchestra.run.smithi049.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-18T14:14:42.155 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:42.155 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:14:42.428 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-mgr-cephadm 2024-01-18T14:14:43.013 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:20 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:14:43.174 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:14:43.175 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================= 2024-01-18T14:14:43.175 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-01-18T14:14:43.175 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================= 2024-01-18T14:14:43.175 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-01-18T14:14:43.175 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-cephadm noarch 2:19.0.0-749.g45e21eee.el9 ceph-noarch 133 k 2024-01-18T14:14:43.175 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-01-18T14:14:43.175 INFO:teuthology.orchestra.run.smithi049.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-18T14:14:43.175 INFO:teuthology.orchestra.run.smithi049.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-18T14:14:43.175 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-18T14:14:43.175 INFO:teuthology.orchestra.run.smithi049.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-18T14:14:43.175 INFO:teuthology.orchestra.run.smithi049.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-18T14:14:43.175 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:43.175 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-01-18T14:14:43.176 INFO:teuthology.orchestra.run.smithi049.stdout:======================================================================================= 2024-01-18T14:14:43.176 INFO:teuthology.orchestra.run.smithi049.stdout:Install 6 Packages 2024-01-18T14:14:43.176 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:43.179 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 6.8 M 2024-01-18T14:14:43.179 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 33 M 2024-01-18T14:14:43.179 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-01-18T14:14:43.445 INFO:teuthology.orchestra.run.smithi049.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.2 MB/s | 228 kB 00:00 2024-01-18T14:14:43.587 INFO:teuthology.orchestra.run.smithi049.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-01-18T14:14:43.687 INFO:teuthology.orchestra.run.smithi049.stdout:(3/6): ceph-mgr-cephadm-19.0.0-749.g45e21eee.el 312 kB/s | 133 kB 00:00 2024-01-18T14:14:43.745 INFO:teuthology.orchestra.run.smithi049.stdout:(4/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2024-01-18T14:14:43.762 INFO:teuthology.orchestra.run.smithi049.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 329 kB/s | 58 kB 00:00 2024-01-18T14:14:43.871 INFO:teuthology.orchestra.run.smithi049.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 277 kB/s | 51 kB 00:00 2024-01-18T14:14:43.872 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-01-18T14:14:43.873 INFO:teuthology.orchestra.run.smithi049.stdout:Total 9.7 MB/s | 6.8 MB 00:00 2024-01-18T14:14:43.873 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-01-18T14:14:43.882 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-01-18T14:14:43.882 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-01-18T14:14:43.977 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-01-18T14:14:43.983 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-01-18T14:14:44.289 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-01-18T14:14:44.470 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-01-18T14:14:44.524 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-01-18T14:14:45.083 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-01-18T14:14:45.187 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-01-18T14:14:45.255 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-01-18T14:14:45.285 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-749.g45e21eee.el9.noarch 6/6 2024-01-18T14:14:46.085 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-749.g45e21eee.el9.noarch 6/6 2024-01-18T14:14:46.085 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-749.g45e21eee.el9.noarch 1/6 2024-01-18T14:14:46.085 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-01-18T14:14:46.085 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-01-18T14:14:46.085 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-01-18T14:14:46.085 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-01-18T14:14:46.414 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-01-18T14:14:46.415 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:46.415 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-01-18T14:14:46.415 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-mgr-cephadm-2:19.0.0-749.g45e21eee.el9.noarch 2024-01-18T14:14:46.415 INFO:teuthology.orchestra.run.smithi049.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-18T14:14:46.415 INFO:teuthology.orchestra.run.smithi049.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-18T14:14:46.415 INFO:teuthology.orchestra.run.smithi049.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-18T14:14:46.415 INFO:teuthology.orchestra.run.smithi049.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-18T14:14:46.415 INFO:teuthology.orchestra.run.smithi049.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-18T14:14:46.415 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:46.415 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:14:46.682 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-fuse 2024-01-18T14:14:47.270 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:24 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:14:47.427 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:14:47.427 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:14:47.427 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-01-18T14:14:47.427 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:14:47.427 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-01-18T14:14:47.428 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse x86_64 2:19.0.0-749.g45e21eee.el9 ceph 858 k 2024-01-18T14:14:47.428 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-01-18T14:14:47.428 INFO:teuthology.orchestra.run.smithi049.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-18T14:14:47.428 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:47.428 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-01-18T14:14:47.428 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:14:47.428 INFO:teuthology.orchestra.run.smithi049.stdout:Install 2 Packages 2024-01-18T14:14:47.428 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:47.429 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 937 k 2024-01-18T14:14:47.429 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 3.1 M 2024-01-18T14:14:47.429 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-01-18T14:14:47.581 INFO:teuthology.orchestra.run.smithi049.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 518 kB/s | 79 kB 00:00 2024-01-18T14:14:48.514 INFO:teuthology.orchestra.run.smithi049.stdout:(2/2): ceph-fuse-19.0.0-749.g45e21eee.el9.x86_6 791 kB/s | 858 kB 00:01 2024-01-18T14:14:48.516 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-01-18T14:14:48.516 INFO:teuthology.orchestra.run.smithi049.stdout:Total 861 kB/s | 937 kB 00:01 2024-01-18T14:14:48.516 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-01-18T14:14:48.539 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-01-18T14:14:48.539 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-01-18T14:14:48.620 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-01-18T14:14:48.621 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-01-18T14:14:48.818 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-01-18T14:14:48.963 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-01-18T14:14:49.034 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-fuse-2:19.0.0-749.g45e21eee.el9.x86_64 2/2 2024-01-18T14:14:49.729 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-fuse-2:19.0.0-749.g45e21eee.el9.x86_64 2/2 2024-01-18T14:14:49.729 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-fuse-2:19.0.0-749.g45e21eee.el9.x86_64 1/2 2024-01-18T14:14:50.032 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-01-18T14:14:50.032 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:50.032 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-01-18T14:14:50.032 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-fuse-2:19.0.0-749.g45e21eee.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-01-18T14:14:50.032 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:50.032 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:14:50.311 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install ceph-volume 2024-01-18T14:14:50.901 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:27 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:14:51.061 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:14:51.062 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:14:51.062 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-01-18T14:14:51.062 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:14:51.062 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-01-18T14:14:51.063 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-volume noarch 2:19.0.0-749.g45e21eee.el9 ceph-noarch 262 k 2024-01-18T14:14:51.063 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:51.063 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-01-18T14:14:51.063 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:14:51.063 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-01-18T14:14:51.063 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:51.063 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 262 k 2024-01-18T14:14:51.063 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 1.2 M 2024-01-18T14:14:51.063 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-01-18T14:14:51.342 INFO:teuthology.orchestra.run.smithi049.stdout:ceph-volume-19.0.0-749.g45e21eee.el9.noarch.rpm 941 kB/s | 262 kB 00:00 2024-01-18T14:14:51.343 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-01-18T14:14:51.343 INFO:teuthology.orchestra.run.smithi049.stdout:Total 936 kB/s | 262 kB 00:00 2024-01-18T14:14:51.343 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-01-18T14:14:51.352 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-01-18T14:14:51.352 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-01-18T14:14:51.388 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-01-18T14:14:51.388 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-01-18T14:14:51.621 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-01-18T14:14:51.696 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : ceph-volume-2:19.0.0-749.g45e21eee.el9.noarch 1/1 2024-01-18T14:14:51.712 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: ceph-volume-2:19.0.0-749.g45e21eee.el9.noarch 1/1 2024-01-18T14:14:51.712 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-18T14:14:51.712 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-18T14:14:51.712 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:52.816 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : ceph-volume-2:19.0.0-749.g45e21eee.el9.noarch 1/1 2024-01-18T14:14:52.816 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:52.816 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-01-18T14:14:52.817 INFO:teuthology.orchestra.run.smithi049.stdout: ceph-volume-2:19.0.0-749.g45e21eee.el9.noarch 2024-01-18T14:14:52.817 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:52.817 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:14:53.092 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librados-devel 2024-01-18T14:14:53.684 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:30 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:14:53.843 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:14:53.844 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:14:53.844 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-01-18T14:14:53.844 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:14:53.844 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-01-18T14:14:53.844 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel x86_64 2:19.0.0-749.g45e21eee.el9 ceph 127 k 2024-01-18T14:14:53.844 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:53.844 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-01-18T14:14:53.844 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:14:53.844 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-01-18T14:14:53.844 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:53.845 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 127 k 2024-01-18T14:14:53.845 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 457 k 2024-01-18T14:14:53.845 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-01-18T14:14:54.073 INFO:teuthology.orchestra.run.smithi049.stdout:librados-devel-19.0.0-749.g45e21eee.el9.x86_64. 555 kB/s | 127 kB 00:00 2024-01-18T14:14:54.074 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-01-18T14:14:54.074 INFO:teuthology.orchestra.run.smithi049.stdout:Total 551 kB/s | 127 kB 00:00 2024-01-18T14:14:54.074 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-01-18T14:14:54.080 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-01-18T14:14:54.080 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-01-18T14:14:54.138 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-01-18T14:14:54.138 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-01-18T14:14:54.289 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-01-18T14:14:54.382 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librados-devel-2:19.0.0-749.g45e21eee.el9.x86_64 1/1 2024-01-18T14:14:54.829 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: librados-devel-2:19.0.0-749.g45e21eee.el9.x86_64 1/1 2024-01-18T14:14:55.120 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librados-devel-2:19.0.0-749.g45e21eee.el9.x86_64 1/1 2024-01-18T14:14:55.120 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:55.121 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-01-18T14:14:55.121 INFO:teuthology.orchestra.run.smithi049.stdout: librados-devel-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:14:55.121 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:55.121 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:14:55.398 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install libcephfs2 2024-01-18T14:14:55.984 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:32 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:14:56.089 INFO:teuthology.orchestra.run.smithi049.stdout:Package libcephfs2-2:19.0.0-749.g45e21eee.el9.x86_64 is already installed. 2024-01-18T14:14:56.143 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:14:56.144 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-01-18T14:14:56.144 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:14:56.217 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install libcephfs-devel 2024-01-18T14:14:56.807 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:33 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:14:56.967 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:14:56.967 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:14:56.967 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-01-18T14:14:56.967 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:14:56.968 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-01-18T14:14:56.968 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel x86_64 2:19.0.0-749.g45e21eee.el9 ceph 31 k 2024-01-18T14:14:56.968 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:56.968 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-01-18T14:14:56.968 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:14:56.968 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-01-18T14:14:56.968 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:56.968 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 31 k 2024-01-18T14:14:56.968 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 140 k 2024-01-18T14:14:56.968 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-01-18T14:14:57.143 INFO:teuthology.orchestra.run.smithi049.stdout:libcephfs-devel-19.0.0-749.g45e21eee.el9.x86_64 178 kB/s | 31 kB 00:00 2024-01-18T14:14:57.144 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-01-18T14:14:57.144 INFO:teuthology.orchestra.run.smithi049.stdout:Total 177 kB/s | 31 kB 00:00 2024-01-18T14:14:57.144 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-01-18T14:14:57.147 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-01-18T14:14:57.147 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-01-18T14:14:57.179 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-01-18T14:14:57.180 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-01-18T14:14:57.269 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-01-18T14:14:57.377 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libcephfs-devel-2:19.0.0-749.g45e21eee.el9.x86_64 1/1 2024-01-18T14:14:57.768 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-749.g45e21eee.el9.x86_64 1/1 2024-01-18T14:14:58.046 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libcephfs-devel-2:19.0.0-749.g45e21eee.el9.x86_64 1/1 2024-01-18T14:14:58.046 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:58.046 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-01-18T14:14:58.046 INFO:teuthology.orchestra.run.smithi049.stdout: libcephfs-devel-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:14:58.046 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:14:58.046 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:14:58.237 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librados2 2024-01-18T14:14:58.824 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:35 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:14:58.927 INFO:teuthology.orchestra.run.smithi049.stdout:Package librados2-2:19.0.0-749.g45e21eee.el9.x86_64 is already installed. 2024-01-18T14:14:58.983 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:14:58.984 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-01-18T14:14:58.984 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:14:59.058 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install librbd1 2024-01-18T14:14:59.644 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:36 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:14:59.747 INFO:teuthology.orchestra.run.smithi049.stdout:Package librbd1-2:19.0.0-749.g45e21eee.el9.x86_64 is already installed. 2024-01-18T14:14:59.801 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:14:59.802 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-01-18T14:14:59.802 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:14:59.876 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rados 2024-01-18T14:15:00.457 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:37 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:15:00.561 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rados-2:19.0.0-749.g45e21eee.el9.x86_64 is already installed. 2024-01-18T14:15:00.615 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:15:00.616 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-01-18T14:15:00.616 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:15:00.693 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rgw 2024-01-18T14:15:01.277 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:38 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:15:01.380 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rgw-2:19.0.0-749.g45e21eee.el9.x86_64 is already installed. 2024-01-18T14:15:01.434 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:15:01.435 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-01-18T14:15:01.435 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:15:01.514 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-cephfs 2024-01-18T14:15:02.101 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:39 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:15:02.205 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-cephfs-2:19.0.0-749.g45e21eee.el9.x86_64 is already installed. 2024-01-18T14:15:02.260 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:15:02.261 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-01-18T14:15:02.261 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:15:02.330 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-rbd 2024-01-18T14:15:02.918 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:39 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:15:03.023 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-rbd-2:19.0.0-749.g45e21eee.el9.x86_64 is already installed. 2024-01-18T14:15:03.080 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:15:03.081 INFO:teuthology.orchestra.run.smithi049.stdout:Nothing to do. 2024-01-18T14:15:03.081 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:15:03.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-fuse 2024-01-18T14:15:03.741 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:40 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:15:03.899 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:15:03.900 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:15:03.900 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-01-18T14:15:03.900 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:15:03.900 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-01-18T14:15:03.900 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse x86_64 2:19.0.0-749.g45e21eee.el9 ceph 86 k 2024-01-18T14:15:03.900 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:03.901 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-01-18T14:15:03.901 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:15:03.901 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-01-18T14:15:03.901 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:03.901 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 86 k 2024-01-18T14:15:03.901 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 256 k 2024-01-18T14:15:03.901 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-01-18T14:15:04.412 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-fuse-19.0.0-749.g45e21eee.el9.x86_64.rpm 168 kB/s | 86 kB 00:00 2024-01-18T14:15:04.413 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-01-18T14:15:04.414 INFO:teuthology.orchestra.run.smithi049.stdout:Total 167 kB/s | 86 kB 00:00 2024-01-18T14:15:04.414 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-01-18T14:15:04.421 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-01-18T14:15:04.421 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-01-18T14:15:04.481 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-01-18T14:15:04.481 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-01-18T14:15:04.627 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-01-18T14:15:04.747 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-fuse-2:19.0.0-749.g45e21eee.el9.x86_64 1/1 2024-01-18T14:15:05.164 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-fuse-2:19.0.0-749.g45e21eee.el9.x86_64 1/1 2024-01-18T14:15:05.485 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-fuse-2:19.0.0-749.g45e21eee.el9.x86_64 1/1 2024-01-18T14:15:05.485 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:05.485 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-01-18T14:15:05.485 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-fuse-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:15:05.485 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:05.485 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:15:05.716 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-mirror 2024-01-18T14:15:06.305 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:43 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:15:06.466 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:15:06.466 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:15:06.466 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repo Size 2024-01-18T14:15:06.467 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:15:06.467 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-01-18T14:15:06.467 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror x86_64 2:19.0.0-749.g45e21eee.el9 ceph 3.0 M 2024-01-18T14:15:06.467 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:06.467 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-01-18T14:15:06.467 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:15:06.467 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-01-18T14:15:06.467 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:06.467 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 3.0 M 2024-01-18T14:15:06.467 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 14 M 2024-01-18T14:15:06.467 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-01-18T14:15:07.157 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-mirror-19.0.0-749.g45e21eee.el9.x86_64.rpm 4.4 MB/s | 3.0 MB 00:00 2024-01-18T14:15:07.157 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-01-18T14:15:07.158 INFO:teuthology.orchestra.run.smithi049.stdout:Total 4.4 MB/s | 3.0 MB 00:00 2024-01-18T14:15:07.158 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-01-18T14:15:07.169 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-01-18T14:15:07.169 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-01-18T14:15:07.247 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-01-18T14:15:07.247 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-01-18T14:15:07.699 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-01-18T14:15:07.766 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-mirror-2:19.0.0-749.g45e21eee.el9.x86_64 1/1 2024-01-18T14:15:07.792 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-mirror-2:19.0.0-749.g45e21eee.el9.x86_64 1/1 2024-01-18T14:15:07.792 INFO:teuthology.orchestra.run.smithi049.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-18T14:15:07.792 INFO:teuthology.orchestra.run.smithi049.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-18T14:15:07.792 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-18T14:15:07.792 INFO:teuthology.orchestra.run.smithi049.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-18T14:15:07.792 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:09.035 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-mirror-2:19.0.0-749.g45e21eee.el9.x86_64 1/1 2024-01-18T14:15:09.035 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:09.035 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-01-18T14:15:09.035 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-mirror-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:15:09.036 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:09.036 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:15:09.329 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install rbd-nbd 2024-01-18T14:15:09.921 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:02:46 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:15:10.077 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:15:10.078 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:15:10.078 INFO:teuthology.orchestra.run.smithi049.stdout: Package Architecture Version Repository Size 2024-01-18T14:15:10.078 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:15:10.078 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-01-18T14:15:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd x86_64 2:19.0.0-749.g45e21eee.el9 ceph 169 k 2024-01-18T14:15:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:10.079 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-01-18T14:15:10.079 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:15:10.079 INFO:teuthology.orchestra.run.smithi049.stdout:Install 1 Package 2024-01-18T14:15:10.079 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:10.079 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 169 k 2024-01-18T14:15:10.079 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 536 k 2024-01-18T14:15:10.079 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-01-18T14:15:10.310 INFO:teuthology.orchestra.run.smithi049.stdout:rbd-nbd-19.0.0-749.g45e21eee.el9.x86_64.rpm 732 kB/s | 169 kB 00:00 2024-01-18T14:15:10.311 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-01-18T14:15:10.311 INFO:teuthology.orchestra.run.smithi049.stdout:Total 727 kB/s | 169 kB 00:00 2024-01-18T14:15:10.311 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-01-18T14:15:10.320 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-01-18T14:15:10.320 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-01-18T14:15:10.379 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-01-18T14:15:10.379 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-01-18T14:15:10.543 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-01-18T14:15:10.637 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : rbd-nbd-2:19.0.0-749.g45e21eee.el9.x86_64 1/1 2024-01-18T14:15:11.150 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: rbd-nbd-2:19.0.0-749.g45e21eee.el9.x86_64 1/1 2024-01-18T14:15:11.444 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : rbd-nbd-2:19.0.0-749.g45e21eee.el9.x86_64 1/1 2024-01-18T14:15:11.444 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:11.444 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-01-18T14:15:11.445 INFO:teuthology.orchestra.run.smithi049.stdout: rbd-nbd-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:15:11.445 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:11.445 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:15:11.674 DEBUG:teuthology.parallel:result is None 2024-01-18T14:15:11.675 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=45e21eee52b813a77f38a534981b5dd2d1bbb6e3 2024-01-18T14:15:11.901 DEBUG:teuthology.orchestra.run.smithi049:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-18T14:15:11.922 INFO:teuthology.orchestra.run.smithi049.stdout:19.0.0-749.g45e21eee.el9 2024-01-18T14:15:11.923 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-749.g45e21eee.el9 2024-01-18T14:15:11.923 INFO:teuthology.task.install:The correct ceph version 19.0.0-749.g45e21eee is installed. 2024-01-18T14:15:11.925 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-18T14:15:11.925 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-01-18T14:15:11.925 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-18T14:15:11.995 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-18T14:15:11.996 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-01-18T14:15:11.996 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/daemon-helper 2024-01-18T14:15:12.067 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-18T14:15:12.137 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-18T14:15:12.138 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-01-18T14:15:12.138 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-18T14:15:12.208 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-18T14:15:12.280 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-18T14:15:12.280 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-01-18T14:15:12.280 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/usr/bin/stdin-killer 2024-01-18T14:15:12.351 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-18T14:15:12.422 INFO:teuthology.run_tasks:Running task ceph... 2024-01-18T14:15:12.504 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-18T14:15:12.504 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 777 /var/log/ceph 2024-01-18T14:15:12.535 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-18T14:15:12.535 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-18T14:15:12.605 INFO:tasks.ceph:Creating extra log directories... 2024-01-18T14:15:12.606 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-18T14:15:12.685 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-18T14:15:12.685 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-18T14:15:12.686 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2024-01-18_13:55:25-crimson-rados-wip-matanb-crimson-testing-18.1-crimson-only-distro-crimson-smithi/7520293', 'branch': 'wip-matanb-crimson-testing-18.1-crimson-only', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read}', 'email': None, 'first_in_suite': False, 'job_id': '7520293', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2024-01-18_13:55:25-crimson-rados-wip-matanb-crimson-testing-18.1-crimson-only-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-testing-18.1-crimson-only'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '45e21eee52b813a77f38a534981b5dd2d1bbb6e3'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '45e21eee52b813a77f38a534981b5dd2d1bbb6e3'}}, 'workunit': {'branch': 'wip-matanb-crimson-testing-18.1-crimson-only', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '45e21eee52b813a77f38a534981b5dd2d1bbb6e3'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 8851, 'sha1': '45e21eee52b813a77f38a534981b5dd2d1bbb6e3', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-matanb-crimson-testing-18.1-crimson-only', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_45e21eee52b813a77f38a534981b5dd2d1bbb6e3/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '45e21eee52b813a77f38a534981b5dd2d1bbb6e3', 'targets': {'smithi049.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAeJhrp6kQZlsvIaMgSXIYTQ1r1l9fOVudiEfd9DWwGBB5EZwMT5QoU9QzRIMnXLFGCnNnO1FKzG/SGVW/EUxuc='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '45e21eee52b813a77f38a534981b5dd2d1bbb6e3', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randread'], 'norandommap': True, 'op_size': [4096], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2]}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'cd45576300487d997e5a85abed65500b9f5d143b', 'timestamp': '2024-01-18_13:55:25', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-18T14:15:12.686 DEBUG:teuthology.orchestra.run.smithi049:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-18T14:15:12.733 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -d -m0777 -- /var/run/ceph 2024-01-18T14:15:12.804 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-01-18T14:15:12.804 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/scratch_devs of=/dev/stdout 2024-01-18T14:15:12.860 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-18T14:15:12.860 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_1 2024-01-18T14:15:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-18T14:15:12.915 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-18T14:15:12.916 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-01-18T14:15:12.916 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-18T14:15:12.916 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-01-18T14:15:12.916 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-01-18 14:15:08.055848065 +0000 2024-01-18T14:15:12.916 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-01-18 14:11:30.064805777 +0000 2024-01-18T14:15:12.916 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-01-18 14:11:30.064805777 +0000 2024-01-18T14:15:12.916 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-01-18 14:11:30.064805777 +0000 2024-01-18T14:15:12.916 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-18T14:15:12.985 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-01-18T14:15:12.985 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-01-18T14:15:12.985 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 0.000129034 s, 4.0 MB/s 2024-01-18T14:15:12.986 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-18T14:15:13.042 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_2 2024-01-18T14:15:13.097 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-18T14:15:13.097 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-18T14:15:13.097 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-01-18T14:15:13.097 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-18T14:15:13.097 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-01-18T14:15:13.097 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-01-18 14:15:08.056848037 +0000 2024-01-18T14:15:13.097 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-01-18 14:11:30.407796157 +0000 2024-01-18T14:15:13.098 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-01-18 14:11:30.407796157 +0000 2024-01-18T14:15:13.098 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-01-18 14:11:30.407796157 +0000 2024-01-18T14:15:13.098 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-18T14:15:13.166 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-01-18T14:15:13.166 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-01-18T14:15:13.166 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 7.9508e-05 s, 6.4 MB/s 2024-01-18T14:15:13.167 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-18T14:15:13.223 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_3 2024-01-18T14:15:13.279 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-18T14:15:13.279 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-18T14:15:13.279 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-01-18T14:15:13.279 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-18T14:15:13.279 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-01-18T14:15:13.279 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-01-18 14:15:08.056848037 +0000 2024-01-18T14:15:13.279 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-01-18 14:11:30.743786734 +0000 2024-01-18T14:15:13.279 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-01-18 14:11:30.743786734 +0000 2024-01-18T14:15:13.280 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-01-18 14:11:30.743786734 +0000 2024-01-18T14:15:13.280 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-18T14:15:13.348 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-01-18T14:15:13.348 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-01-18T14:15:13.348 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 9.7766e-05 s, 5.2 MB/s 2024-01-18T14:15:13.350 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-18T14:15:13.406 DEBUG:teuthology.orchestra.run.smithi049:> stat /dev/vg_nvme/lv_4 2024-01-18T14:15:13.461 INFO:teuthology.orchestra.run.smithi049.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-18T14:15:13.461 INFO:teuthology.orchestra.run.smithi049.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-18T14:15:13.461 INFO:teuthology.orchestra.run.smithi049.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-01-18T14:15:13.461 INFO:teuthology.orchestra.run.smithi049.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-18T14:15:13.462 INFO:teuthology.orchestra.run.smithi049.stdout:Context: system_u:object_r:device_t:s0 2024-01-18T14:15:13.462 INFO:teuthology.orchestra.run.smithi049.stdout:Access: 2024-01-18 14:15:08.057848009 +0000 2024-01-18T14:15:13.462 INFO:teuthology.orchestra.run.smithi049.stdout:Modify: 2024-01-18 14:11:31.080777282 +0000 2024-01-18T14:15:13.462 INFO:teuthology.orchestra.run.smithi049.stdout:Change: 2024-01-18 14:11:31.080777282 +0000 2024-01-18T14:15:13.462 INFO:teuthology.orchestra.run.smithi049.stdout: Birth: 2024-01-18 14:11:31.080777282 +0000 2024-01-18T14:15:13.462 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-18T14:15:13.530 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records in 2024-01-18T14:15:13.530 INFO:teuthology.orchestra.run.smithi049.stderr:1+0 records out 2024-01-18T14:15:13.530 INFO:teuthology.orchestra.run.smithi049.stderr:512 bytes copied, 9.2482e-05 s, 5.5 MB/s 2024-01-18T14:15:13.532 DEBUG:teuthology.orchestra.run.smithi049:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-18T14:15:13.588 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-18T14:15:13.588 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi049.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-01-18T14:15:13.588 INFO:tasks.ceph:Generating config... 2024-01-18T14:15:13.590 INFO:tasks.ceph:[osd] debug monc = 20 2024-01-18T14:15:13.590 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-01-18T14:15:13.590 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-01-18T14:15:13.591 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-01-18T14:15:13.591 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-01-18T14:15:13.591 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-01-18T14:15:13.591 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2024-01-18T14:15:13.591 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-01-18T14:15:13.591 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-01-18T14:15:13.591 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-01-18T14:15:13.591 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-01-18T14:15:13.591 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-01-18T14:15:13.591 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-01-18T14:15:13.591 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-01-18T14:15:13.591 INFO:tasks.ceph:[osd] err to stderr = True 2024-01-18T14:15:13.591 INFO:tasks.ceph:[osd] log flush on exit = True 2024-01-18T14:15:13.591 INFO:tasks.ceph:[osd] log to file = False 2024-01-18T14:15:13.591 INFO:tasks.ceph:[osd] log to stderr = True 2024-01-18T14:15:13.592 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-01-18T14:15:13.592 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-01-18T14:15:13.592 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-01-18T14:15:13.592 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-01-18T14:15:13.592 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-01-18T14:15:13.592 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-01-18T14:15:13.592 INFO:tasks.ceph:[global] auth client required = none 2024-01-18T14:15:13.592 INFO:tasks.ceph:[global] auth cluster required = none 2024-01-18T14:15:13.592 INFO:tasks.ceph:[global] auth service required = none 2024-01-18T14:15:13.592 INFO:tasks.ceph:[global] auth supported = none 2024-01-18T14:15:13.592 INFO:tasks.ceph:[global] debug asok = 0/0 2024-01-18T14:15:13.592 INFO:tasks.ceph:[global] debug auth = 0/0 2024-01-18T14:15:13.592 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-01-18T14:15:13.592 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-01-18T14:15:13.593 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-01-18T14:15:13.593 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-01-18T14:15:13.593 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-01-18T14:15:13.593 INFO:tasks.ceph:[global] debug client = 0/0 2024-01-18T14:15:13.593 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-01-18T14:15:13.593 INFO:tasks.ceph:[global] debug context = 0/0 2024-01-18T14:15:13.593 INFO:tasks.ceph:[global] debug crush = 0/0 2024-01-18T14:15:13.593 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-01-18T14:15:13.593 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-01-18T14:15:13.593 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-01-18T14:15:13.593 INFO:tasks.ceph:[global] debug filer = 0/0 2024-01-18T14:15:13.593 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-01-18T14:15:13.593 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-01-18T14:15:13.593 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-01-18T14:15:13.594 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-01-18T14:15:13.594 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-01-18T14:15:13.594 INFO:tasks.ceph:[global] debug journal = 0/0 2024-01-18T14:15:13.594 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-01-18T14:15:13.594 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-01-18T14:15:13.594 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-01-18T14:15:13.594 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-01-18T14:15:13.594 INFO:tasks.ceph:[global] debug mds = 0/0 2024-01-18T14:15:13.594 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-01-18T14:15:13.594 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-01-18T14:15:13.594 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-01-18T14:15:13.594 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-01-18T14:15:13.594 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-01-18T14:15:13.594 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-01-18T14:15:13.594 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-01-18T14:15:13.595 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-01-18T14:15:13.595 INFO:tasks.ceph:[global] debug mon = 0/0 2024-01-18T14:15:13.595 INFO:tasks.ceph:[global] debug monc = 0/0 2024-01-18T14:15:13.595 INFO:tasks.ceph:[global] debug ms = 0/0 2024-01-18T14:15:13.595 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-01-18T14:15:13.595 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-01-18T14:15:13.595 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-01-18T14:15:13.595 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-01-18T14:15:13.595 INFO:tasks.ceph:[global] debug osd = 0/0 2024-01-18T14:15:13.595 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-01-18T14:15:13.595 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-01-18T14:15:13.595 INFO:tasks.ceph:[global] debug rados = 0/0 2024-01-18T14:15:13.595 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-01-18T14:15:13.595 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-01-18T14:15:13.595 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-01-18T14:15:13.595 INFO:tasks.ceph:[global] debug refs = 0/0 2024-01-18T14:15:13.596 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-01-18T14:15:13.596 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-01-18T14:15:13.596 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-01-18T14:15:13.596 INFO:tasks.ceph:[global] debug striper = 0/0 2024-01-18T14:15:13.596 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-01-18T14:15:13.596 INFO:tasks.ceph:[global] debug timer = 0/0 2024-01-18T14:15:13.596 INFO:tasks.ceph:[global] debug tp = 0/0 2024-01-18T14:15:13.596 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-01-18T14:15:13.596 INFO:tasks.ceph:[global] ms client mode = crc 2024-01-18T14:15:13.596 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-01-18T14:15:13.596 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-01-18T14:15:13.596 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-01-18T14:15:13.596 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-01-18T14:15:13.596 INFO:tasks.ceph:[global] ms service mode = crc 2024-01-18T14:15:13.596 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-18T14:15:13.597 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-18T14:15:13.597 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-01-18T14:15:13.597 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-01-18T14:15:13.597 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-01-18T14:15:13.597 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-01-18T14:15:13.597 INFO:tasks.ceph:Setting up mon.a... 2024-01-18T14:15:13.597 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-18T14:15:13.692 INFO:teuthology.orchestra.run.smithi049.stdout:creating /etc/ceph/ceph.keyring 2024-01-18T14:15:13.695 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-18T14:15:13.749 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-18T14:15:13.779 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.49')] 2024-01-18T14:15:13.780 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.49', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2024-01-18T14:15:13.781 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-01-18T14:15:13.781 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-18T14:15:13.834 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.49 --print /home/ubuntu/cephtest/ceph.monmap 2024-01-18T14:15:13.920 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:13.918+0000 7f0b15255e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:13.920 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:13.918+0000 7f0b15255e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:13.920 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:13.918+0000 7f0b15255e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:13.921 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-18T14:15:13.921 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool: generated fsid 2ed1de91-5c4c-40c7-8c2f-096ad2527fde 2024-01-18T14:15:13.921 INFO:teuthology.orchestra.run.smithi049.stdout:setting min_mon_release = quincy 2024-01-18T14:15:13.921 INFO:teuthology.orchestra.run.smithi049.stdout:epoch 0 2024-01-18T14:15:13.921 INFO:teuthology.orchestra.run.smithi049.stdout:fsid 2ed1de91-5c4c-40c7-8c2f-096ad2527fde 2024-01-18T14:15:13.921 INFO:teuthology.orchestra.run.smithi049.stdout:last_changed 2024-01-18T14:15:13.920347+0000 2024-01-18T14:15:13.921 INFO:teuthology.orchestra.run.smithi049.stdout:created 2024-01-18T14:15:13.920347+0000 2024-01-18T14:15:13.921 INFO:teuthology.orchestra.run.smithi049.stdout:min_mon_release 17 (quincy) 2024-01-18T14:15:13.921 INFO:teuthology.orchestra.run.smithi049.stdout:election_strategy: 1 2024-01-18T14:15:13.921 INFO:teuthology.orchestra.run.smithi049.stdout:0: [v2:172.21.15.49:3300/0,v1:172.21.15.49:6789/0] mon.a 2024-01-18T14:15:13.921 INFO:teuthology.orchestra.run.smithi049.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-01-18T14:15:13.922 DEBUG:teuthology.orchestra.run.smithi049:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-18T14:15:13.937 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 2ed1de91-5c4c-40c7-8c2f-096ad2527fde... 2024-01-18T14:15:13.938 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-18T14:15:14.038 INFO:teuthology.orchestra.run.smithi049.stdout:[global] 2024-01-18T14:15:14.038 INFO:teuthology.orchestra.run.smithi049.stdout: chdir = "" 2024-01-18T14:15:14.039 INFO:teuthology.orchestra.run.smithi049.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-18T14:15:14.039 INFO:teuthology.orchestra.run.smithi049.stdout: auth supported = none 2024-01-18T14:15:14.039 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.039 INFO:teuthology.orchestra.run.smithi049.stdout: filestore xattr use omap = true 2024-01-18T14:15:14.039 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.039 INFO:teuthology.orchestra.run.smithi049.stdout: mon clock drift allowed = 1.000 2024-01-18T14:15:14.039 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.039 INFO:teuthology.orchestra.run.smithi049.stdout: osd crush chooseleaf type = 0 2024-01-18T14:15:14.039 INFO:teuthology.orchestra.run.smithi049.stdout: auth debug = true 2024-01-18T14:15:14.039 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.039 INFO:teuthology.orchestra.run.smithi049.stdout: ms die on old message = true 2024-01-18T14:15:14.039 INFO:teuthology.orchestra.run.smithi049.stdout: ms die on bug = true 2024-01-18T14:15:14.039 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.040 INFO:teuthology.orchestra.run.smithi049.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-18T14:15:14.040 INFO:teuthology.orchestra.run.smithi049.stdout: mon pg warn max object skew = 0 2024-01-18T14:15:14.040 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.040 INFO:teuthology.orchestra.run.smithi049.stdout: # disable pg_autoscaler by default for new pools 2024-01-18T14:15:14.040 INFO:teuthology.orchestra.run.smithi049.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-18T14:15:14.040 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.040 INFO:teuthology.orchestra.run.smithi049.stdout: osd pool default size = 2 2024-01-18T14:15:14.040 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.040 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd allow primary affinity = true 2024-01-18T14:15:14.040 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd allow pg remap = true 2024-01-18T14:15:14.040 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on legacy crush tunables = false 2024-01-18T14:15:14.040 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on crush straw calc version zero = false 2024-01-18T14:15:14.040 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on no sortbitwise = false 2024-01-18T14:15:14.040 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on osd down out interval zero = false 2024-01-18T14:15:14.040 INFO:teuthology.orchestra.run.smithi049.stdout: mon warn on too few osds = false 2024-01-18T14:15:14.041 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-18T14:15:14.041 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-18T14:15:14.041 INFO:teuthology.orchestra.run.smithi049.stdout: mon_allow_pool_size_one = true 2024-01-18T14:15:14.041 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.041 INFO:teuthology.orchestra.run.smithi049.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-18T14:15:14.041 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.041 INFO:teuthology.orchestra.run.smithi049.stdout: osd default data pool replay window = 5 2024-01-18T14:15:14.041 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.041 INFO:teuthology.orchestra.run.smithi049.stdout: mon allow pool delete = true 2024-01-18T14:15:14.041 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.041 INFO:teuthology.orchestra.run.smithi049.stdout: mon cluster log file level = debug 2024-01-18T14:15:14.041 INFO:teuthology.orchestra.run.smithi049.stdout: debug asserts on shutdown = true 2024-01-18T14:15:14.041 INFO:teuthology.orchestra.run.smithi049.stdout: mon health detail to clog = false 2024-01-18T14:15:14.041 INFO:teuthology.orchestra.run.smithi049.stdout: mon host = 172.21.15.49 2024-01-18T14:15:14.041 INFO:teuthology.orchestra.run.smithi049.stdout: auth client required = none 2024-01-18T14:15:14.041 INFO:teuthology.orchestra.run.smithi049.stdout: auth cluster required = none 2024-01-18T14:15:14.042 INFO:teuthology.orchestra.run.smithi049.stdout: auth service required = none 2024-01-18T14:15:14.042 INFO:teuthology.orchestra.run.smithi049.stdout: debug asok = 0/0 2024-01-18T14:15:14.042 INFO:teuthology.orchestra.run.smithi049.stdout: debug auth = 0/0 2024-01-18T14:15:14.042 INFO:teuthology.orchestra.run.smithi049.stdout: debug bdev = 0/0 2024-01-18T14:15:14.042 INFO:teuthology.orchestra.run.smithi049.stdout: debug bluefs = 0/0 2024-01-18T14:15:14.042 INFO:teuthology.orchestra.run.smithi049.stdout: debug bluestore = 0/0 2024-01-18T14:15:14.042 INFO:teuthology.orchestra.run.smithi049.stdout: debug buffer = 0/0 2024-01-18T14:15:14.042 INFO:teuthology.orchestra.run.smithi049.stdout: debug civetweb = 0/0 2024-01-18T14:15:14.042 INFO:teuthology.orchestra.run.smithi049.stdout: debug client = 0/0 2024-01-18T14:15:14.042 INFO:teuthology.orchestra.run.smithi049.stdout: debug compressor = 0/0 2024-01-18T14:15:14.042 INFO:teuthology.orchestra.run.smithi049.stdout: debug context = 0/0 2024-01-18T14:15:14.042 INFO:teuthology.orchestra.run.smithi049.stdout: debug crush = 0/0 2024-01-18T14:15:14.042 INFO:teuthology.orchestra.run.smithi049.stdout: debug crypto = 0/0 2024-01-18T14:15:14.042 INFO:teuthology.orchestra.run.smithi049.stdout: debug dpdk = 0/0 2024-01-18T14:15:14.043 INFO:teuthology.orchestra.run.smithi049.stdout: debug eventtrace = 0/0 2024-01-18T14:15:14.043 INFO:teuthology.orchestra.run.smithi049.stdout: debug filer = 0/0 2024-01-18T14:15:14.043 INFO:teuthology.orchestra.run.smithi049.stdout: debug filestore = 0/0 2024-01-18T14:15:14.043 INFO:teuthology.orchestra.run.smithi049.stdout: debug finisher = 0/0 2024-01-18T14:15:14.043 INFO:teuthology.orchestra.run.smithi049.stdout: debug fuse = 0/0 2024-01-18T14:15:14.043 INFO:teuthology.orchestra.run.smithi049.stdout: debug heartbeatmap = 0/0 2024-01-18T14:15:14.043 INFO:teuthology.orchestra.run.smithi049.stdout: debug javaclient = 0/0 2024-01-18T14:15:14.043 INFO:teuthology.orchestra.run.smithi049.stdout: debug journal = 0/0 2024-01-18T14:15:14.043 INFO:teuthology.orchestra.run.smithi049.stdout: debug journaler = 0/0 2024-01-18T14:15:14.043 INFO:teuthology.orchestra.run.smithi049.stdout: debug kstore = 0/0 2024-01-18T14:15:14.043 INFO:teuthology.orchestra.run.smithi049.stdout: debug leveldb = 0/0 2024-01-18T14:15:14.043 INFO:teuthology.orchestra.run.smithi049.stdout: debug lockdep = 0/0 2024-01-18T14:15:14.043 INFO:teuthology.orchestra.run.smithi049.stdout: debug mds = 0/0 2024-01-18T14:15:14.043 INFO:teuthology.orchestra.run.smithi049.stdout: debug mds balancer = 0/0 2024-01-18T14:15:14.043 INFO:teuthology.orchestra.run.smithi049.stdout: debug mds locker = 0/0 2024-01-18T14:15:14.044 INFO:teuthology.orchestra.run.smithi049.stdout: debug mds log = 0/0 2024-01-18T14:15:14.044 INFO:teuthology.orchestra.run.smithi049.stdout: debug mds log expire = 0/0 2024-01-18T14:15:14.044 INFO:teuthology.orchestra.run.smithi049.stdout: debug mds migrator = 0/0 2024-01-18T14:15:14.044 INFO:teuthology.orchestra.run.smithi049.stdout: debug memdb = 0/0 2024-01-18T14:15:14.044 INFO:teuthology.orchestra.run.smithi049.stdout: debug mgr = 0/0 2024-01-18T14:15:14.044 INFO:teuthology.orchestra.run.smithi049.stdout: debug mgrc = 0/0 2024-01-18T14:15:14.044 INFO:teuthology.orchestra.run.smithi049.stdout: debug mon = 0/0 2024-01-18T14:15:14.044 INFO:teuthology.orchestra.run.smithi049.stdout: debug monc = 0/0 2024-01-18T14:15:14.044 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 0/0 2024-01-18T14:15:14.044 INFO:teuthology.orchestra.run.smithi049.stdout: debug objclass = 0/0 2024-01-18T14:15:14.044 INFO:teuthology.orchestra.run.smithi049.stdout: debug objectcacher = 0/0 2024-01-18T14:15:14.044 INFO:teuthology.orchestra.run.smithi049.stdout: debug objecter = 0/0 2024-01-18T14:15:14.044 INFO:teuthology.orchestra.run.smithi049.stdout: debug optracker = 0/0 2024-01-18T14:15:14.044 INFO:teuthology.orchestra.run.smithi049.stdout: debug osd = 0/0 2024-01-18T14:15:14.044 INFO:teuthology.orchestra.run.smithi049.stdout: debug paxos = 0/0 2024-01-18T14:15:14.045 INFO:teuthology.orchestra.run.smithi049.stdout: debug perfcounter = 0/0 2024-01-18T14:15:14.045 INFO:teuthology.orchestra.run.smithi049.stdout: debug rados = 0/0 2024-01-18T14:15:14.045 INFO:teuthology.orchestra.run.smithi049.stdout: debug rbd = 0/0 2024-01-18T14:15:14.045 INFO:teuthology.orchestra.run.smithi049.stdout: debug rbd mirror = 0/0 2024-01-18T14:15:14.045 INFO:teuthology.orchestra.run.smithi049.stdout: debug rbd replay = 0/0 2024-01-18T14:15:14.045 INFO:teuthology.orchestra.run.smithi049.stdout: debug refs = 0/0 2024-01-18T14:15:14.045 INFO:teuthology.orchestra.run.smithi049.stdout: debug rgw = 0/0 2024-01-18T14:15:14.045 INFO:teuthology.orchestra.run.smithi049.stdout: debug rgw sync = 0/0 2024-01-18T14:15:14.045 INFO:teuthology.orchestra.run.smithi049.stdout: debug rocksdb = 0/0 2024-01-18T14:15:14.045 INFO:teuthology.orchestra.run.smithi049.stdout: debug striper = 0/0 2024-01-18T14:15:14.045 INFO:teuthology.orchestra.run.smithi049.stdout: debug throttle = 0/0 2024-01-18T14:15:14.045 INFO:teuthology.orchestra.run.smithi049.stdout: debug timer = 0/0 2024-01-18T14:15:14.045 INFO:teuthology.orchestra.run.smithi049.stdout: debug tp = 0/0 2024-01-18T14:15:14.045 INFO:teuthology.orchestra.run.smithi049.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-01-18T14:15:14.045 INFO:teuthology.orchestra.run.smithi049.stdout: ms client mode = crc 2024-01-18T14:15:14.046 INFO:teuthology.orchestra.run.smithi049.stdout: ms cluster mode = crc 2024-01-18T14:15:14.046 INFO:teuthology.orchestra.run.smithi049.stdout: ms mon client mode = crc 2024-01-18T14:15:14.046 INFO:teuthology.orchestra.run.smithi049.stdout: ms mon cluster mode = crc 2024-01-18T14:15:14.046 INFO:teuthology.orchestra.run.smithi049.stdout: ms mon service mode = crc 2024-01-18T14:15:14.046 INFO:teuthology.orchestra.run.smithi049.stdout: ms service mode = crc 2024-01-18T14:15:14.046 INFO:teuthology.orchestra.run.smithi049.stdout: fsid = 2ed1de91-5c4c-40c7-8c2f-096ad2527fde 2024-01-18T14:15:14.046 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.046 INFO:teuthology.orchestra.run.smithi049.stdout:[osd] 2024-01-18T14:15:14.046 INFO:teuthology.orchestra.run.smithi049.stdout: osd journal size = 100 2024-01-18T14:15:14.046 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.046 INFO:teuthology.orchestra.run.smithi049.stdout: osd scrub load threshold = 5.0 2024-01-18T14:15:14.046 INFO:teuthology.orchestra.run.smithi049.stdout: osd scrub max interval = 600 2024-01-18T14:15:14.046 INFO:teuthology.orchestra.run.smithi049.stdout: osd mclock profile = high_recovery_ops 2024-01-18T14:15:14.046 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.046 INFO:teuthology.orchestra.run.smithi049.stdout: osd recover clone overlap = true 2024-01-18T14:15:14.047 INFO:teuthology.orchestra.run.smithi049.stdout: osd recovery max chunk = 1048576 2024-01-18T14:15:14.047 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.047 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug shutdown = true 2024-01-18T14:15:14.047 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug op order = true 2024-01-18T14:15:14.047 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug verify stray on activate = true 2024-01-18T14:15:14.047 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug trim objects = true 2024-01-18T14:15:14.047 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.047 INFO:teuthology.orchestra.run.smithi049.stdout: osd open classes on start = true 2024-01-18T14:15:14.047 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug pg log writeout = true 2024-01-18T14:15:14.047 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.047 INFO:teuthology.orchestra.run.smithi049.stdout: osd deep scrub update digest min age = 30 2024-01-18T14:15:14.047 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.047 INFO:teuthology.orchestra.run.smithi049.stdout: osd map max advance = 10 2024-01-18T14:15:14.047 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.047 INFO:teuthology.orchestra.run.smithi049.stdout: journal zero on create = true 2024-01-18T14:15:14.047 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.048 INFO:teuthology.orchestra.run.smithi049.stdout: filestore ondisk finisher threads = 3 2024-01-18T14:15:14.048 INFO:teuthology.orchestra.run.smithi049.stdout: filestore apply finisher threads = 3 2024-01-18T14:15:14.048 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.048 INFO:teuthology.orchestra.run.smithi049.stdout: bdev debug aio = true 2024-01-18T14:15:14.048 INFO:teuthology.orchestra.run.smithi049.stdout: osd debug misdirected ops = true 2024-01-18T14:15:14.048 INFO:teuthology.orchestra.run.smithi049.stdout: debug monc = 20 2024-01-18T14:15:14.048 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore block size = 96636764160 2024-01-18T14:15:14.048 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore compression algorithm = snappy 2024-01-18T14:15:14.048 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore compression mode = aggressive 2024-01-18T14:15:14.048 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore fsck on mount = True 2024-01-18T14:15:14.048 INFO:teuthology.orchestra.run.smithi049.stdout: bluestore rocksdb cf = False 2024-01-18T14:15:14.048 INFO:teuthology.orchestra.run.smithi049.stdout: crimson seastar smp = 3 2024-01-18T14:15:14.048 INFO:teuthology.orchestra.run.smithi049.stdout: debug bluefs = 20 2024-01-18T14:15:14.048 INFO:teuthology.orchestra.run.smithi049.stdout: debug bluestore = 20 2024-01-18T14:15:14.048 INFO:teuthology.orchestra.run.smithi049.stdout: debug filestore = 0/0 2024-01-18T14:15:14.049 INFO:teuthology.orchestra.run.smithi049.stdout: debug journal = 0/0 2024-01-18T14:15:14.049 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 0/0 2024-01-18T14:15:14.049 INFO:teuthology.orchestra.run.smithi049.stdout: debug osd = 0/0 2024-01-18T14:15:14.049 INFO:teuthology.orchestra.run.smithi049.stdout: debug rocksdb = 10 2024-01-18T14:15:14.049 INFO:teuthology.orchestra.run.smithi049.stdout: err to stderr = True 2024-01-18T14:15:14.049 INFO:teuthology.orchestra.run.smithi049.stdout: log flush on exit = True 2024-01-18T14:15:14.049 INFO:teuthology.orchestra.run.smithi049.stdout: log to file = False 2024-01-18T14:15:14.049 INFO:teuthology.orchestra.run.smithi049.stdout: log to stderr = True 2024-01-18T14:15:14.049 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd backfillfull_ratio = 0.85 2024-01-18T14:15:14.049 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd full ratio = 0.9 2024-01-18T14:15:14.049 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd nearfull ratio = 0.8 2024-01-18T14:15:14.049 INFO:teuthology.orchestra.run.smithi049.stdout: osd failsafe full ratio = 0.95 2024-01-18T14:15:14.049 INFO:teuthology.orchestra.run.smithi049.stdout: osd objectstore = bluestore 2024-01-18T14:15:14.049 INFO:teuthology.orchestra.run.smithi049.stdout: osd shutdown pgref assert = True 2024-01-18T14:15:14.049 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.050 INFO:teuthology.orchestra.run.smithi049.stdout:[mgr] 2024-01-18T14:15:14.050 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 1 2024-01-18T14:15:14.050 INFO:teuthology.orchestra.run.smithi049.stdout: debug mgr = 20 2024-01-18T14:15:14.050 INFO:teuthology.orchestra.run.smithi049.stdout: debug mon = 20 2024-01-18T14:15:14.050 INFO:teuthology.orchestra.run.smithi049.stdout: debug auth = 20 2024-01-18T14:15:14.050 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min pgs per osd = 4 2024-01-18T14:15:14.050 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min bytes per osd = 10 2024-01-18T14:15:14.050 INFO:teuthology.orchestra.run.smithi049.stdout: mgr/telemetry/nag = false 2024-01-18T14:15:14.050 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.050 INFO:teuthology.orchestra.run.smithi049.stdout:[mon] 2024-01-18T14:15:14.050 INFO:teuthology.orchestra.run.smithi049.stdout: debug ms = 0/0 2024-01-18T14:15:14.050 INFO:teuthology.orchestra.run.smithi049.stdout: debug mon = 0/0 2024-01-18T14:15:14.050 INFO:teuthology.orchestra.run.smithi049.stdout: debug paxos = 0/0 2024-01-18T14:15:14.050 INFO:teuthology.orchestra.run.smithi049.stdout: debug auth = 20 2024-01-18T14:15:14.050 INFO:teuthology.orchestra.run.smithi049.stdout: mon data avail warn = 5 2024-01-18T14:15:14.050 INFO:teuthology.orchestra.run.smithi049.stdout: mon mgr mkfs grace = 240 2024-01-18T14:15:14.051 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min pgs per osd = 4 2024-01-18T14:15:14.051 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd reporter subtree level = osd 2024-01-18T14:15:14.051 INFO:teuthology.orchestra.run.smithi049.stdout: mon osd prime pg temp = true 2024-01-18T14:15:14.051 INFO:teuthology.orchestra.run.smithi049.stdout: mon reweight min bytes per osd = 10 2024-01-18T14:15:14.051 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.051 INFO:teuthology.orchestra.run.smithi049.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-18T14:15:14.051 INFO:teuthology.orchestra.run.smithi049.stdout: auth mon ticket ttl = 660 # 11m 2024-01-18T14:15:14.051 INFO:teuthology.orchestra.run.smithi049.stdout: auth service ticket ttl = 240 # 4m 2024-01-18T14:15:14.051 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.051 INFO:teuthology.orchestra.run.smithi049.stdout: # don't complain about insecure global_id in the test suite 2024-01-18T14:15:14.051 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-18T14:15:14.051 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-18T14:15:14.051 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.051 INFO:teuthology.orchestra.run.smithi049.stdout: # 1m isn't quite enough 2024-01-18T14:15:14.051 INFO:teuthology.orchestra.run.smithi049.stdout: mon_down_mkfs_grace = 2m 2024-01-18T14:15:14.052 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.052 INFO:teuthology.orchestra.run.smithi049.stdout: mon_warn_on_filestore_osds = false 2024-01-18T14:15:14.052 INFO:teuthology.orchestra.run.smithi049.stdout: osd pool default crimson = True 2024-01-18T14:15:14.052 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:14.052 INFO:teuthology.orchestra.run.smithi049.stdout:[client] 2024-01-18T14:15:14.052 INFO:teuthology.orchestra.run.smithi049.stdout: rgw cache enabled = true 2024-01-18T14:15:14.052 INFO:teuthology.orchestra.run.smithi049.stdout: rgw enable ops log = true 2024-01-18T14:15:14.052 INFO:teuthology.orchestra.run.smithi049.stdout: rgw enable usage log = true 2024-01-18T14:15:14.052 INFO:teuthology.orchestra.run.smithi049.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-18T14:15:14.052 INFO:teuthology.orchestra.run.smithi049.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-18T14:15:14.052 INFO:teuthology.orchestra.run.smithi049.stdout:[mon.a] 2024-01-18T14:15:14.055 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-18T14:15:14.056 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-18T14:15:14.161 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-18T14:15:14.162 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-01-18T14:15:14.162 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-18T14:15:14.177 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-01-18T14:15:14.177 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-18T14:15:14.233 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:15:14.233 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-01-18T14:15:14.233 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-18T14:15:14.233 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-18T14:15:14.319 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-01-18T14:15:14.320 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-18T14:15:14.374 INFO:tasks.ceph:Setting up mon nodes... 2024-01-18T14:15:14.375 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-18T14:15:14.375 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-18T14:15:14.482 INFO:teuthology.orchestra.run.smithi049.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-18T14:15:14.484 INFO:tasks.ceph:Setting up mds nodes... 2024-01-18T14:15:14.485 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-18T14:15:14.485 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-18T14:15:14.537 INFO:teuthology.orchestra.run.smithi049.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-18T14:15:14.556 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-18T14:15:14.556 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi049.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-01-18T14:15:14.556 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-18T14:15:14.586 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-18T14:15:14.587 INFO:tasks.ceph:role: osd.0 2024-01-18T14:15:14.587 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:15:14.587 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-18T14:15:14.657 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-18T14:15:14.657 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-18T14:15:14.658 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-18T14:15:14.658 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-18T14:15:14.658 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-18T14:15:14.658 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2024-01-18T14:15:14.658 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-18T14:15:14.658 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-18T14:15:14.658 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-18T14:15:14.658 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-18T14:15:14.708 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2024-01-18T14:15:14.710 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2024-01-18T14:15:14.710 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-18T14:15:14.786 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-18T14:15:14.862 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-18T14:15:14.933 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-18T14:15:14.933 INFO:tasks.ceph:role: osd.1 2024-01-18T14:15:14.933 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:15:14.933 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-18T14:15:15.003 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-18T14:15:15.003 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-18T14:15:15.003 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-18T14:15:15.003 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-18T14:15:15.003 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-18T14:15:15.003 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2024-01-18T14:15:15.003 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-18T14:15:15.004 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-18T14:15:15.004 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-18T14:15:15.004 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-18T14:15:15.042 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2024-01-18T14:15:15.044 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2024-01-18T14:15:15.044 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-18T14:15:15.128 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-18T14:15:15.163 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-18T14:15:15.234 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-18T14:15:15.234 INFO:tasks.ceph:role: osd.2 2024-01-18T14:15:15.234 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:15:15.234 DEBUG:teuthology.orchestra.run.smithi049:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-18T14:15:15.304 INFO:teuthology.orchestra.run.smithi049.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-18T14:15:15.304 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-18T14:15:15.304 INFO:teuthology.orchestra.run.smithi049.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-18T14:15:15.304 INFO:teuthology.orchestra.run.smithi049.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-18T14:15:15.304 INFO:teuthology.orchestra.run.smithi049.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-18T14:15:15.304 INFO:teuthology.orchestra.run.smithi049.stdout: = sunit=0 swidth=0 blks 2024-01-18T14:15:15.304 INFO:teuthology.orchestra.run.smithi049.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-18T14:15:15.304 INFO:teuthology.orchestra.run.smithi049.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-18T14:15:15.304 INFO:teuthology.orchestra.run.smithi049.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-18T14:15:15.304 INFO:teuthology.orchestra.run.smithi049.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-18T14:15:15.342 INFO:teuthology.orchestra.run.smithi049.stdout:Discarding blocks...Done. 2024-01-18T14:15:15.344 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi049.front.sepia.ceph.com -o noatime 2024-01-18T14:15:15.344 DEBUG:teuthology.orchestra.run.smithi049:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-18T14:15:15.419 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-18T14:15:15.495 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-18T14:15:16.075 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-18T14:15:16.076 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:16,076 seastar - Reactor backend: linux-aio 2024-01-18T14:15:16.092 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:16,092 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-18T14:15:16.092 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:16,092 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-18T14:15:16.092 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:16,092 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-18T14:15:16.092 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:16,092 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-18T14:15:16.094 INFO:teuthology.orchestra.run.smithi049.stderr:==28236==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-18T14:15:16.100 INFO:teuthology.orchestra.run.smithi049.stderr:==28236==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe9761b000; bottom 0x7f1a1c1cd000; size: 0x00e47b44e000 (981320654848) 2024-01-18T14:15:16.100 INFO:teuthology.orchestra.run.smithi049.stderr:False positive error reports may follow 2024-01-18T14:15:16.100 INFO:teuthology.orchestra.run.smithi049.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-18T14:15:16.663 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2024-01-18T14:15:16.663 INFO:teuthology.orchestra.run.smithi049.stderr:Suppressions used: 2024-01-18T14:15:16.663 INFO:teuthology.orchestra.run.smithi049.stderr: count bytes template 2024-01-18T14:15:16.663 INFO:teuthology.orchestra.run.smithi049.stderr: 1 8 MallocExtension::Register 2024-01-18T14:15:16.663 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2024-01-18T14:15:16.663 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-01-18T14:15:16.719 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-18T14:15:16.719 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:16,720 seastar - Reactor backend: linux-aio 2024-01-18T14:15:16.747 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:16,747 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-18T14:15:16.747 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:16,747 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-18T14:15:16.747 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:16,747 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-18T14:15:16.754 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:16,754 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-18T14:15:16.757 INFO:teuthology.orchestra.run.smithi049.stderr:==28234==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-18T14:15:16.763 INFO:teuthology.orchestra.run.smithi049.stderr:==28234==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe9761b000; bottom 0x7f1a19fb6000; size: 0x00e47d665000 (981356400640) 2024-01-18T14:15:16.763 INFO:teuthology.orchestra.run.smithi049.stderr:False positive error reports may follow 2024-01-18T14:15:16.763 INFO:teuthology.orchestra.run.smithi049.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-18T14:15:16.789 INFO:teuthology.orchestra.run.smithi049.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2e8329f4 0x2e833da2 0x2e8352d4 0x2e835658 0x2e8357a2 0x2e835bf8 0x54daf 0x1f73094b 0x1f852288 0x23f8194e 0x23f86f8a 0x25eb43b4 0x25ebb432 0x25ebdd7f 0x25ebef44 0x23ac9fcb 0x23ae6f32 0x23b5b5a3 0x2e7a53ad 0x2e8085de 0x2eaae775 0x2eab05bf 0x2e461892 0x2e46379b 0x1f71e8c8 0x3feaf 0x3ff5f 0x1f2dda44 2024-01-18T14:15:16.789 INFO:teuthology.orchestra.run.smithi049.stderr:kernel callstack: 2024-01-18T14:15:20.788 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2024-01-18T14:15:20.789 INFO:teuthology.orchestra.run.smithi049.stderr:Suppressions used: 2024-01-18T14:15:20.789 INFO:teuthology.orchestra.run.smithi049.stderr: count bytes template 2024-01-18T14:15:20.789 INFO:teuthology.orchestra.run.smithi049.stderr: 1 8 MallocExtension::Register 2024-01-18T14:15:20.789 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2024-01-18T14:15:20.789 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-01-18T14:15:20.789 INFO:teuthology.orchestra.run.smithi049.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 2ed1de91-5c4c-40c7-8c2f-096ad2527fde 2024-01-18T14:15:20.828 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-18T14:15:20.859 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-18T14:15:21.439 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-18T14:15:21.440 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:21,440 seastar - Reactor backend: linux-aio 2024-01-18T14:15:21.456 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:21,456 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-18T14:15:21.456 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:21,456 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-18T14:15:21.456 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:21,456 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-18T14:15:21.456 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:21,456 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-18T14:15:21.459 INFO:teuthology.orchestra.run.smithi049.stderr:==28418==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-18T14:15:21.464 INFO:teuthology.orchestra.run.smithi049.stderr:==28418==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffdbc5df000; bottom 0x7f9833901000; size: 0x006588cde000 (436086890496) 2024-01-18T14:15:21.464 INFO:teuthology.orchestra.run.smithi049.stderr:False positive error reports may follow 2024-01-18T14:15:21.464 INFO:teuthology.orchestra.run.smithi049.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-18T14:15:21.490 INFO:teuthology.orchestra.run.smithi049.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2e8329f4 0x2e833da2 0x2e8352d4 0x2e835658 0x2e8357a2 0x2e835bf8 0x54daf 0x24149753 0x2414a006 0x2415cf31 0x2415d90c 0x241a7ee6 0x24172d5c 0x2417695c 0x24176ec3 0x241a35fd 0x241a3e69 0x2415775d 0x24166baa 0x24168289 0x24168c5e 0x24169d24 0x2417eb5a 0x2417f2eb 0x2418121a 0x24182c1e 0x2418397c 0x24143deb 0x2414345a 0x23fec281 0x25eb38fe 0x25ebb432 0x25ebdd7f 0x25ebef44 0x23ac9fcb 0x23ae6f32 0x23b5b5a3 0x2e7a53ad 0x2e8085de 0x2eaae775 0x2eab05bf 0x2e461892 0x2e46379b 0x1f93f161 0x1f945c09 0x1f71d012 0x3feaf 0x3ff5f 0x1f2dda44 2024-01-18T14:15:21.490 INFO:teuthology.orchestra.run.smithi049.stderr:kernel callstack: 2024-01-18T14:15:22.007 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2024-01-18T14:15:22.008 INFO:teuthology.orchestra.run.smithi049.stderr:Suppressions used: 2024-01-18T14:15:22.008 INFO:teuthology.orchestra.run.smithi049.stderr: count bytes template 2024-01-18T14:15:22.008 INFO:teuthology.orchestra.run.smithi049.stderr: 1 8 MallocExtension::Register 2024-01-18T14:15:22.008 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2024-01-18T14:15:22.008 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-01-18T14:15:22.064 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-18T14:15:22.064 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:22,065 seastar - Reactor backend: linux-aio 2024-01-18T14:15:22.092 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:22,092 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-18T14:15:22.092 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:22,092 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-18T14:15:22.092 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:22,092 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-18T14:15:22.100 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:22,100 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-18T14:15:22.102 INFO:teuthology.orchestra.run.smithi049.stderr:==28416==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-18T14:15:22.108 INFO:teuthology.orchestra.run.smithi049.stderr:==28416==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffdbc5df000; bottom 0x7f98316ea000; size: 0x00658aef5000 (436122636288) 2024-01-18T14:15:22.109 INFO:teuthology.orchestra.run.smithi049.stderr:False positive error reports may follow 2024-01-18T14:15:22.109 INFO:teuthology.orchestra.run.smithi049.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-18T14:15:26.142 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2024-01-18T14:15:26.142 INFO:teuthology.orchestra.run.smithi049.stderr:Suppressions used: 2024-01-18T14:15:26.143 INFO:teuthology.orchestra.run.smithi049.stderr: count bytes template 2024-01-18T14:15:26.143 INFO:teuthology.orchestra.run.smithi049.stderr: 1 8 MallocExtension::Register 2024-01-18T14:15:26.143 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2024-01-18T14:15:26.143 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-01-18T14:15:26.143 INFO:teuthology.orchestra.run.smithi049.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 2ed1de91-5c4c-40c7-8c2f-096ad2527fde 2024-01-18T14:15:26.183 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-18T14:15:26.213 DEBUG:teuthology.orchestra.run.smithi049:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-18T14:15:26.777 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-18T14:15:26.778 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:26,778 seastar - Reactor backend: linux-aio 2024-01-18T14:15:26.798 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:26,798 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-18T14:15:26.798 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:26,798 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-18T14:15:26.798 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:26,798 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-18T14:15:26.798 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:26,798 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-18T14:15:26.800 INFO:teuthology.orchestra.run.smithi049.stderr:==28601==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-18T14:15:26.806 INFO:teuthology.orchestra.run.smithi049.stderr:==28601==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe63f7c000; bottom 0x7f3085440000; size: 0x00cddeb3c000 (884204617728) 2024-01-18T14:15:26.806 INFO:teuthology.orchestra.run.smithi049.stderr:False positive error reports may follow 2024-01-18T14:15:26.807 INFO:teuthology.orchestra.run.smithi049.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-18T14:15:27.335 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2024-01-18T14:15:27.335 INFO:teuthology.orchestra.run.smithi049.stderr:Suppressions used: 2024-01-18T14:15:27.335 INFO:teuthology.orchestra.run.smithi049.stderr: count bytes template 2024-01-18T14:15:27.335 INFO:teuthology.orchestra.run.smithi049.stderr: 1 8 MallocExtension::Register 2024-01-18T14:15:27.335 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2024-01-18T14:15:27.336 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-01-18T14:15:27.403 INFO:teuthology.orchestra.run.smithi049.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-18T14:15:27.403 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:27,403 seastar - Reactor backend: linux-aio 2024-01-18T14:15:27.420 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:27,420 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-18T14:15:27.421 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:27,420 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-18T14:15:27.421 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:27,420 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-18T14:15:27.428 INFO:teuthology.orchestra.run.smithi049.stderr:INFO 2024-01-18 14:15:27,428 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-18T14:15:27.431 INFO:teuthology.orchestra.run.smithi049.stderr:==28599==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-18T14:15:27.437 INFO:teuthology.orchestra.run.smithi049.stderr:==28599==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe63f7c000; bottom 0x7f3083229000; size: 0x00cde0d53000 (884240363520) 2024-01-18T14:15:27.437 INFO:teuthology.orchestra.run.smithi049.stderr:False positive error reports may follow 2024-01-18T14:15:27.437 INFO:teuthology.orchestra.run.smithi049.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-18T14:15:27.463 INFO:teuthology.orchestra.run.smithi049.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2e8329f4 0x2e833da2 0x2e8352d4 0x2e835658 0x2e8357a2 0x2e835bf8 0x54daf 0xb94bf 0x94469 0x94d46 0x23e2e8d7 0x23e2ebff 0x23ee3ad3 0x23ee6ce3 0x23f7c727 0x23f81919 0x23f86f8a 0x25eb43b4 0x25ebb432 0x25ebdd7f 0x25ebef44 0x23ac9fcb 0x23ae6f32 0x23b5b5a3 0x2e7a53ad 0x2e8085de 0x2eaae775 0x2eab05bf 0x2e461892 0x2e46379b 0x1f71e8c8 0x3feaf 0x3ff5f 0x1f2dda44 2024-01-18T14:15:27.463 INFO:teuthology.orchestra.run.smithi049.stderr:kernel callstack: 2024-01-18T14:15:31.442 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2024-01-18T14:15:31.442 INFO:teuthology.orchestra.run.smithi049.stderr:Suppressions used: 2024-01-18T14:15:31.442 INFO:teuthology.orchestra.run.smithi049.stderr: count bytes template 2024-01-18T14:15:31.442 INFO:teuthology.orchestra.run.smithi049.stderr: 1 8 MallocExtension::Register 2024-01-18T14:15:31.442 INFO:teuthology.orchestra.run.smithi049.stderr:----------------------------------------------------- 2024-01-18T14:15:31.443 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-01-18T14:15:31.443 INFO:teuthology.orchestra.run.smithi049.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 2ed1de91-5c4c-40c7-8c2f-096ad2527fde 2024-01-18T14:15:31.482 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-18T14:15:31.513 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-18T14:15:31.513 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-01-18T14:15:31.513 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-18T14:15:31.584 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-01-18T14:15:31.584 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-18T14:15:31.654 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-01-18T14:15:31.655 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-18T14:15:31.726 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-01-18T14:15:31.727 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-18T14:15:31.797 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-01-18T14:15:31.797 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-18T14:15:31.852 INFO:tasks.ceph:Adding keys to all mons... 2024-01-18T14:15:31.852 DEBUG:teuthology.orchestra.run.smithi049:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-18T14:15:31.921 INFO:teuthology.orchestra.run.smithi049.stdout:[mgr.x] 2024-01-18T14:15:31.921 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQByMqllhea+HBAA44YVA+rCfr/z82C3dtA1fA== 2024-01-18T14:15:31.921 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.0] 2024-01-18T14:15:31.921 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQB0Mqllz9ZuMBAAGm0AhhvngargvjOr+2A8Eg== 2024-01-18T14:15:31.922 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.1] 2024-01-18T14:15:31.922 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQB6Mqllgn73CRAAPp0MZn0fPi8rK6fHc2/W3w== 2024-01-18T14:15:31.922 INFO:teuthology.orchestra.run.smithi049.stdout:[osd.2] 2024-01-18T14:15:31.922 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQB/MqllAKv+HBAAaskDCfGkLQLogqHvPW0Ipw== 2024-01-18T14:15:31.922 INFO:teuthology.orchestra.run.smithi049.stdout:[client.0] 2024-01-18T14:15:31.922 INFO:teuthology.orchestra.run.smithi049.stdout: key = AQByMqllV1cFIBAAxgME9JdNHzajZAHtitiWPg== 2024-01-18T14:15:31.923 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-18T14:15:32.025 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-18T14:15:32.087 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-18T14:15:32.149 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-18T14:15:32.211 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-18T14:15:32.276 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-18T14:15:32.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-18T14:15:32.306 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-18T14:15:32.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:32.410+0000 7f054665b9c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:32.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:32.410+0000 7f054665b9c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:32.422 DEBUG:teuthology.orchestra.run.smithi049:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-18T14:15:32.453 DEBUG:teuthology.orchestra.run.smithi049:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-18T14:15:32.508 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-18T14:15:32.508 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-18T14:15:32.509 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-01-18T14:15:32.550 INFO:tasks.ceph.mon.a:Started 2024-01-18T14:15:32.550 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-18T14:15:32.550 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-18T14:15:32.551 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-01-18T14:15:32.552 INFO:tasks.ceph.mgr.x:Started 2024-01-18T14:15:32.553 INFO:tasks.ceph:Setting crush tunables to default 2024-01-18T14:15:32.553 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd crush tunables default 2024-01-18T14:15:32.638 INFO:tasks.ceph.mon.a.smithi049.stderr:2024-01-18T14:15:32.637+0000 7f5d4e0109c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:32.638 INFO:tasks.ceph.mon.a.smithi049.stderr:2024-01-18T14:15:32.637+0000 7f5d4e0109c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:32.638 INFO:tasks.ceph.mon.a.smithi049.stderr:2024-01-18T14:15:32.637+0000 7f5d4e0109c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:32.638 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:32.638+0000 7f1de4769f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:32.670 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:32.665+0000 7f4b056f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:32.684 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:32.684+0000 7f4b056f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:32.755 INFO:teuthology.orchestra.run.smithi049.stderr:adjusted tunables profile to default 2024-01-18T14:15:32.766 INFO:tasks.ceph:check_enable_crimson: True 2024-01-18T14:15:32.766 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-01-18T14:15:32.766 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-01-18T14:15:32.842 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:32.841+0000 7f1de4769f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:32.842 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:32.842+0000 7f1de4769f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:32.882 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:32.878+0000 7f5a5079e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:32.884 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:32.884+0000 7f5a5079e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:32.965 INFO:tasks.ceph.mon.a.smithi049.stderr:2024-01-18T14:15:32.965+0000 7f5d45ff1640 -1 WARNING: experimental feature 'crimson' is enabled 2024-01-18T14:15:32.965 INFO:tasks.ceph.mon.a.smithi049.stderr:Please be aware that this feature is experimental, untested, 2024-01-18T14:15:32.965 INFO:tasks.ceph.mon.a.smithi049.stderr:unsupported, and may result in data corruption, data loss, 2024-01-18T14:15:32.966 INFO:tasks.ceph.mon.a.smithi049.stderr:and/or irreparable damage to your cluster. Do not use 2024-01-18T14:15:32.966 INFO:tasks.ceph.mon.a.smithi049.stderr:feature with important data. 2024-01-18T14:15:32.966 INFO:tasks.ceph.mon.a.smithi049.stderr: 2024-01-18T14:15:33.326 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:33.325+0000 7f1de4769f00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-18T14:15:33.381 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:33.381+0000 7f1de4769f00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-18T14:15:33.767 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-18T14:15:33.767 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-01-18T14:15:33.767 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-18T14:15:33.798 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-01-18T14:15:33.798 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-18T14:15:33.869 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-01-18T14:15:33.869 DEBUG:teuthology.orchestra.run.smithi049:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-18T14:15:33.931 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:33.930+0000 7f1de4769f00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-18T14:15:33.940 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd new efc07a44-0d19-417c-9ea8-e40fcdc36bea 0 2024-01-18T14:15:33.986 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:33.986+0000 7f1de4769f00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-18T14:15:34.034 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:34.034+0000 7f1de4769f00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-18T14:15:34.091 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:34.086+0000 7f20a6d1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:34.092 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:34.092+0000 7f1de4769f00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-18T14:15:34.093 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:34.093+0000 7f20a6d1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:34.137 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:34.137+0000 7f1de4769f00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-18T14:15:34.171 INFO:teuthology.orchestra.run.smithi049.stdout:0 2024-01-18T14:15:34.182 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd new 8432a2a6-e6ee-4b06-ae35-1a5dda778fd3 1 2024-01-18T14:15:34.245 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:34.245+0000 7f1de4769f00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-18T14:15:34.321 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:34.321+0000 7f1de4769f00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-18T14:15:34.333 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:34.328+0000 7f240cd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:34.335 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:34.335+0000 7f240cd9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:34.370 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:34.370+0000 7f1de4769f00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-18T14:15:34.413 INFO:teuthology.orchestra.run.smithi049.stdout:1 2024-01-18T14:15:34.425 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd new 004e99ce-8b31-4946-ad57-027962232501 2 2024-01-18T14:15:34.426 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:34.425+0000 7f1de4769f00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-18T14:15:34.476 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:34.476+0000 7f1de4769f00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-18T14:15:34.536 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:34.531+0000 7f03a235e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:34.538 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:34.538+0000 7f03a235e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:34.617 INFO:teuthology.orchestra.run.smithi049.stdout:2 2024-01-18T14:15:34.628 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-01-18T14:15:34.663 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-18T14:15:34.664 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-01-18T14:15:34.705 INFO:tasks.ceph.osd.0:Started 2024-01-18T14:15:34.706 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-01-18T14:15:34.741 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-18T14:15:34.741 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-01-18T14:15:34.783 INFO:tasks.ceph.osd.1:Started 2024-01-18T14:15:34.784 DEBUG:teuthology.orchestra.run.smithi049:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-01-18T14:15:34.823 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-18T14:15:34.823 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-01-18T14:15:34.866 INFO:tasks.ceph.osd.2:Started 2024-01-18T14:15:34.866 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-18T14:15:34.905 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:34.904+0000 7f1de4769f00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-18T14:15:35.074 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:35.066+0000 7f5f6619e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:35.077 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:35.077+0000 7f5f6619e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:35.125 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:35.125+0000 7f1de4769f00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-18T14:15:35.226 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:35.226 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":6,"fsid":"2ed1de91-5c4c-40c7-8c2f-096ad2527fde","created":"2024-01-18T14:15:32.678265+0000","modified":"2024-01-18T14:15:34.616518+0000","last_up_change":"0.000000","last_in_change":"2024-01-18T14:15:34.616518+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"efc07a44-0d19-417c-9ea8-e40fcdc36bea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8432a2a6-e6ee-4b06-ae35-1a5dda778fd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"004e99ce-8b31-4946-ad57-027962232501","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-18T14:15:35.248 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-18T14:15:35.249 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-18T14:15:35.265 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:35.264+0000 7f1de4769f00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-18T14:15:35.318 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:35.317+0000 7f1de4769f00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-18T14:15:35.428 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:35.428+0000 7f1de4769f00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-18T14:15:35.532 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:35.532+0000 7f1de4769f00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-18T14:15:35.552 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-18T14:15:35.652 INFO:teuthology.misc.health.smithi049.stderr:2024-01-18T14:15:35.648+0000 7f76cc0d5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:35.654 INFO:teuthology.misc.health.smithi049.stderr:2024-01-18T14:15:35.654+0000 7f76cc0d5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:35.688 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:35.688+0000 7f1de4769f00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-18T14:15:35.735 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:35.734+0000 7f1de4769f00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-18T14:15:35.780 INFO:teuthology.misc.health.smithi049.stdout: 2024-01-18T14:15:35.780 INFO:teuthology.misc.health.smithi049.stdout:{"epoch":6,"fsid":"2ed1de91-5c4c-40c7-8c2f-096ad2527fde","created":"2024-01-18T14:15:32.678265+0000","modified":"2024-01-18T14:15:34.616518+0000","last_up_change":"0.000000","last_in_change":"2024-01-18T14:15:34.616518+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"efc07a44-0d19-417c-9ea8-e40fcdc36bea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8432a2a6-e6ee-4b06-ae35-1a5dda778fd3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"004e99ce-8b31-4946-ad57-027962232501","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-18T14:15:35.794 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-01-18T14:15:35.891 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:35.891+0000 7f1de4769f00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-18T14:15:36.777 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:36.776+0000 7f1de4769f00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-18T14:15:37.066 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:37.066+0000 7f1de4769f00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-18T14:15:37.121 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:37.120+0000 7f1de4769f00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-18T14:15:37.228 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:37.227+0000 7f1de4769f00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-18T14:15:38.248 INFO:tasks.ceph.mgr.x.smithi049.stderr:2024-01-18T14:15:38.247+0000 7f1d84430640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-01-18T14:15:42.096 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-18T14:15:42.207 INFO:teuthology.misc.health.smithi049.stderr:2024-01-18T14:15:42.203+0000 7fde48b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:42.209 INFO:teuthology.misc.health.smithi049.stderr:2024-01-18T14:15:42.208+0000 7fde48b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:42.399 INFO:teuthology.misc.health.smithi049.stdout: 2024-01-18T14:15:42.399 INFO:teuthology.misc.health.smithi049.stdout:{"epoch":9,"fsid":"2ed1de91-5c4c-40c7-8c2f-096ad2527fde","created":"2024-01-18T14:15:32.678265+0000","modified":"2024-01-18T14:15:42.252878+0000","last_up_change":"2024-01-18T14:15:40.247704+0000","last_in_change":"2024-01-18T14:15:34.616518+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-18T14:15:41.299600+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"efc07a44-0d19-417c-9ea8-e40fcdc36bea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6803","nonce":3521180995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":3521180995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6809","nonce":3521180995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3521180995}]},"public_addr":"172.21.15.49:6803/3521180995","cluster_addr":"172.21.15.49:6802/3521180995","heartbeat_back_addr":"172.21.15.49:6809/3521180995","heartbeat_front_addr":"172.21.15.49:6808/3521180995","state":["exists","up"]},{"osd":1,"uuid":"8432a2a6-e6ee-4b06-ae35-1a5dda778fd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":47571163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6807","nonce":47571163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":47571163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6813","nonce":47571163}]},"public_addr":"172.21.15.49:6806/47571163","cluster_addr":"172.21.15.49:6807/47571163","heartbeat_back_addr":"172.21.15.49:6812/47571163","heartbeat_front_addr":"172.21.15.49:6813/47571163","state":["exists","up"]},{"osd":2,"uuid":"004e99ce-8b31-4946-ad57-027962232501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":487753380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6805","nonce":487753380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":487753380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6811","nonce":487753380}]},"public_addr":"172.21.15.49:6804/487753380","cluster_addr":"172.21.15.49:6805/487753380","heartbeat_back_addr":"172.21.15.49:6810/487753380","heartbeat_front_addr":"172.21.15.49:6811/487753380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-18T14:15:42.410 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-01-18T14:15:42.410 INFO:tasks.ceph:Creating RBD pool 2024-01-18T14:15:42.410 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd pool create rbd 8 2024-01-18T14:15:42.521 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:42.514+0000 7f62d331e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:42.523 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:42.522+0000 7f62d331e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:43.258 INFO:teuthology.orchestra.run.smithi049.stderr:pool 'rbd' created 2024-01-18T14:15:43.278 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-01-18T14:15:43.405 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:43.395+0000 7f007706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:43.407 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:43.406+0000 7f007706d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:44.261 INFO:teuthology.orchestra.run.smithi049.stderr:enabled application 'rbd' on pool 'rbd' 2024-01-18T14:15:44.275 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-18T14:15:44.276 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-18T14:15:44.276 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-18T14:15:44.276 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-18T14:15:44.276 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-18T14:15:44.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:44.403+0000 7f1a15715640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:44.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:44.408+0000 7f1a15715640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:44.645 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:44.655 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6800","nonce":3495483246},{"type":"v1","addr":"172.21.15.49:6801","nonce":3495483246}]},"active_addr":"172.21.15.49:6801/3495483246","active_change":"2024-01-18T14:15:37.241300+0000","active_mgr_features":4540701272860131327,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":3470244643}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":228291649}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.49:0","nonce":1855573157}]}]} 2024-01-18T14:15:44.659 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-18T14:15:44.659 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-18T14:15:44.659 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-18T14:15:44.770 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:44.766+0000 7f11fde6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:44.772 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:44.771+0000 7f11fde6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:44.959 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:44.959 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":11,"fsid":"2ed1de91-5c4c-40c7-8c2f-096ad2527fde","created":"2024-01-18T14:15:32.678265+0000","modified":"2024-01-18T14:15:44.258898+0000","last_up_change":"2024-01-18T14:15:40.247704+0000","last_in_change":"2024-01-18T14:15:34.616518+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-18T14:15:41.299600+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-18T14:15:42.741300+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"efc07a44-0d19-417c-9ea8-e40fcdc36bea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6803","nonce":3521180995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":3521180995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6809","nonce":3521180995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3521180995}]},"public_addr":"172.21.15.49:6803/3521180995","cluster_addr":"172.21.15.49:6802/3521180995","heartbeat_back_addr":"172.21.15.49:6809/3521180995","heartbeat_front_addr":"172.21.15.49:6808/3521180995","state":["exists","up"]},{"osd":1,"uuid":"8432a2a6-e6ee-4b06-ae35-1a5dda778fd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":47571163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6807","nonce":47571163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":47571163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6813","nonce":47571163}]},"public_addr":"172.21.15.49:6806/47571163","cluster_addr":"172.21.15.49:6807/47571163","heartbeat_back_addr":"172.21.15.49:6812/47571163","heartbeat_front_addr":"172.21.15.49:6813/47571163","state":["exists","up"]},{"osd":2,"uuid":"004e99ce-8b31-4946-ad57-027962232501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":487753380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6805","nonce":487753380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":487753380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6811","nonce":487753380}]},"public_addr":"172.21.15.49:6804/487753380","cluster_addr":"172.21.15.49:6805/487753380","heartbeat_back_addr":"172.21.15.49:6810/487753380","heartbeat_front_addr":"172.21.15.49:6811/487753380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-18T14:15:44.971 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-18T14:15:44.971 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-18T14:15:45.082 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:45.078+0000 7f909fd04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:45.084 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:45.083+0000 7f909fd04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:45.271 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:45.271 INFO:teuthology.orchestra.run.smithi049.stdout:{"epoch":12,"fsid":"2ed1de91-5c4c-40c7-8c2f-096ad2527fde","created":"2024-01-18T14:15:32.678265+0000","modified":"2024-01-18T14:15:45.261898+0000","last_up_change":"2024-01-18T14:15:40.247704+0000","last_in_change":"2024-01-18T14:15:34.616518+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-18T14:15:41.299600+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-18T14:15:42.741300+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"efc07a44-0d19-417c-9ea8-e40fcdc36bea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6803","nonce":3521180995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6802","nonce":3521180995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6809","nonce":3521180995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6808","nonce":3521180995}]},"public_addr":"172.21.15.49:6803/3521180995","cluster_addr":"172.21.15.49:6802/3521180995","heartbeat_back_addr":"172.21.15.49:6809/3521180995","heartbeat_front_addr":"172.21.15.49:6808/3521180995","state":["exists","up"]},{"osd":1,"uuid":"8432a2a6-e6ee-4b06-ae35-1a5dda778fd3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6806","nonce":47571163}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6807","nonce":47571163}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6812","nonce":47571163}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6813","nonce":47571163}]},"public_addr":"172.21.15.49:6806/47571163","cluster_addr":"172.21.15.49:6807/47571163","heartbeat_back_addr":"172.21.15.49:6812/47571163","heartbeat_front_addr":"172.21.15.49:6813/47571163","state":["exists","up"]},{"osd":2,"uuid":"004e99ce-8b31-4946-ad57-027962232501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6804","nonce":487753380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6805","nonce":487753380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6810","nonce":487753380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.49:6811","nonce":487753380}]},"public_addr":"172.21.15.49:6804/487753380","cluster_addr":"172.21.15.49:6805/487753380","heartbeat_back_addr":"172.21.15.49:6810/487753380","heartbeat_front_addr":"172.21.15.49:6811/487753380","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-18T14:15:45.286 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-18T14:15:45.286 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-18T14:15:45.286 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-18T14:15:45.408 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:45.403+0000 7f294d22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:45.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:45.403+0000 7efc7db1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:45.409 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:45.403+0000 7fc89762d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:45.410 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:45.409+0000 7f294d22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:45.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:45.409+0000 7efc7db1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:45.411 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:45.410+0000 7fc89762d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:45.439 INFO:teuthology.orchestra.run.smithi049.stdout:34359738373 2024-01-18T14:15:45.440 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-18T14:15:45.441 INFO:teuthology.orchestra.run.smithi049.stdout:34359738373 2024-01-18T14:15:45.441 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-18T14:15:45.442 INFO:teuthology.orchestra.run.smithi049.stdout:34359738373 2024-01-18T14:15:45.443 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-18T14:15:45.554 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:45.549+0000 7fd638b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:45.556 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:45.555+0000 7fd638b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:45.604 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:45.599+0000 7f145b26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:45.604 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:45.598+0000 7f9aa5b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:45.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:45.605+0000 7f9aa5b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:45.606 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:45.605+0000 7f145b26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:45.757 INFO:teuthology.orchestra.run.smithi049.stdout:34359738372 2024-01-18T14:15:45.774 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.2 2024-01-18T14:15:45.799 INFO:teuthology.orchestra.run.smithi049.stdout:34359738373 2024-01-18T14:15:45.811 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2024-01-18T14:15:45.812 DEBUG:teuthology.parallel:result is None 2024-01-18T14:15:45.812 INFO:teuthology.orchestra.run.smithi049.stdout:34359738372 2024-01-18T14:15:45.825 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2024-01-18T14:15:46.775 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-18T14:15:46.825 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-18T14:15:46.902 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:46.897+0000 7f3ec822d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:46.904 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:46.903+0000 7f3ec822d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:46.954 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:46.949+0000 7f786caad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:46.956 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:46.955+0000 7f786caad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:47.107 INFO:teuthology.orchestra.run.smithi049.stdout:34359738372 2024-01-18T14:15:47.119 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.2 2024-01-18T14:15:47.147 INFO:teuthology.orchestra.run.smithi049.stdout:34359738372 2024-01-18T14:15:47.159 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2024-01-18T14:15:48.120 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-18T14:15:48.159 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-18T14:15:48.250 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:48.243+0000 7f82cc59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:48.253 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:48.251+0000 7f82cc59e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:48.289 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:48.283+0000 7ffbf8c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:48.291 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:48.289+0000 7ffbf8c64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:48.447 INFO:teuthology.orchestra.run.smithi049.stdout:34359738373 2024-01-18T14:15:48.459 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2024-01-18T14:15:48.460 DEBUG:teuthology.parallel:result is None 2024-01-18T14:15:48.481 INFO:teuthology.orchestra.run.smithi049.stdout:34359738373 2024-01-18T14:15:48.493 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2024-01-18T14:15:48.494 DEBUG:teuthology.parallel:result is None 2024-01-18T14:15:48.494 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-18T14:15:48.494 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-18T14:15:48.645 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:48.640+0000 7f7d1651e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:48.646 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:48.645+0000 7f7d1651e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:48.828 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:48.829 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-01-18T14:15:48.840 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-01-18T14:15:47.245725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80908,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034612,"statfs":{"total":289910292480,"available":289827442688,"internally_reserved":0,"allocated":421888,"data_stored":1251088,"data_compressed":59820,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.942015"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-18T14:15:45.301574+0000","last_change":"2024-01-18T14:15:44.385487+0000","last_active":"2024-01-18T14:15:45.301574+0000","last_peered":"2024-01-18T14:15:45.301574+0000","last_clean":"2024-01-18T14:15:45.301574+0000","last_became_active":"2024-01-18T14:15:44.383687+0000","last_became_peered":"2024-01-18T14:15:44.383687+0000","last_unstale":"2024-01-18T14:15:45.301574+0000","last_undegraded":"2024-01-18T14:15:45.301574+0000","last_fullsized":"2024-01-18T14:15:45.301574+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_clean_scrub_stamp":"2024-01-18T14:15:43.255804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-18T14:15:45.301027+0000","last_change":"2024-01-18T14:15:44.369074+0000","last_active":"2024-01-18T14:15:45.301027+0000","last_peered":"2024-01-18T14:15:45.301027+0000","last_clean":"2024-01-18T14:15:45.301027+0000","last_became_active":"2024-01-18T14:15:44.364525+0000","last_became_peered":"2024-01-18T14:15:44.364525+0000","last_unstale":"2024-01-18T14:15:45.301027+0000","last_undegraded":"2024-01-18T14:15:45.301027+0000","last_fullsized":"2024-01-18T14:15:45.301027+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_clean_scrub_stamp":"2024-01-18T14:15:43.255804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-18T14:15:45.298913+0000","last_change":"2024-01-18T14:15:44.366299+0000","last_active":"2024-01-18T14:15:45.298913+0000","last_peered":"2024-01-18T14:15:45.298913+0000","last_clean":"2024-01-18T14:15:45.298913+0000","last_became_active":"2024-01-18T14:15:44.361628+0000","last_became_peered":"2024-01-18T14:15:44.361628+0000","last_unstale":"2024-01-18T14:15:45.298913+0000","last_undegraded":"2024-01-18T14:15:45.298913+0000","last_fullsized":"2024-01-18T14:15:45.298913+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_clean_scrub_stamp":"2024-01-18T14:15:43.255804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-18T14:15:45.288193+0000","last_change":"2024-01-18T14:15:44.389001+0000","last_active":"2024-01-18T14:15:45.288193+0000","last_peered":"2024-01-18T14:15:45.288193+0000","last_clean":"2024-01-18T14:15:45.288193+0000","last_became_active":"2024-01-18T14:15:44.384420+0000","last_became_peered":"2024-01-18T14:15:44.384420+0000","last_unstale":"2024-01-18T14:15:45.288193+0000","last_undegraded":"2024-01-18T14:15:45.288193+0000","last_fullsized":"2024-01-18T14:15:45.288193+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_clean_scrub_stamp":"2024-01-18T14:15:43.255804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-18T14:15:45.303392+0000","last_change":"2024-01-18T14:15:44.355497+0000","last_active":"2024-01-18T14:15:45.303392+0000","last_peered":"2024-01-18T14:15:45.303392+0000","last_clean":"2024-01-18T14:15:45.303392+0000","last_became_active":"2024-01-18T14:15:44.353211+0000","last_became_peered":"2024-01-18T14:15:44.353211+0000","last_unstale":"2024-01-18T14:15:45.303392+0000","last_undegraded":"2024-01-18T14:15:45.303392+0000","last_fullsized":"2024-01-18T14:15:45.303392+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_clean_scrub_stamp":"2024-01-18T14:15:43.255804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-18T14:15:45.272827+0000","last_change":"2024-01-18T14:15:44.358997+0000","last_active":"2024-01-18T14:15:45.272827+0000","last_peered":"2024-01-18T14:15:45.272827+0000","last_clean":"2024-01-18T14:15:45.272827+0000","last_became_active":"2024-01-18T14:15:44.356812+0000","last_became_peered":"2024-01-18T14:15:44.356812+0000","last_unstale":"2024-01-18T14:15:45.272827+0000","last_undegraded":"2024-01-18T14:15:45.272827+0000","last_fullsized":"2024-01-18T14:15:45.272827+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_clean_scrub_stamp":"2024-01-18T14:15:43.255804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-18T14:15:45.274052+0000","last_change":"2024-01-18T14:15:44.347366+0000","last_active":"2024-01-18T14:15:45.274052+0000","last_peered":"2024-01-18T14:15:45.274052+0000","last_clean":"2024-01-18T14:15:45.274052+0000","last_became_active":"2024-01-18T14:15:44.344810+0000","last_became_peered":"2024-01-18T14:15:44.344810+0000","last_unstale":"2024-01-18T14:15:45.274052+0000","last_undegraded":"2024-01-18T14:15:45.274052+0000","last_fullsized":"2024-01-18T14:15:45.274052+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_clean_scrub_stamp":"2024-01-18T14:15:43.255804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-18T14:15:45.292046+0000","last_change":"2024-01-18T14:15:44.333666+0000","last_active":"2024-01-18T14:15:45.292046+0000","last_peered":"2024-01-18T14:15:45.292046+0000","last_clean":"2024-01-18T14:15:45.292046+0000","last_became_active":"2024-01-18T14:15:44.331441+0000","last_became_peered":"2024-01-18T14:15:44.331441+0000","last_unstale":"2024-01-18T14:15:45.292046+0000","last_undegraded":"2024-01-18T14:15:45.292046+0000","last_fullsized":"2024-01-18T14:15:45.292046+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_clean_scrub_stamp":"2024-01-18T14:15:43.255804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-18T14:15:45.303121+0000","last_change":"2024-01-18T14:15:43.329382+0000","last_active":"2024-01-18T14:15:45.303121+0000","last_peered":"2024-01-18T14:15:45.303121+0000","last_clean":"2024-01-18T14:15:45.303121+0000","last_became_active":"2024-01-18T14:15:43.321220+0000","last_became_peered":"2024-01-18T14:15:43.321220+0000","last_unstale":"2024-01-18T14:15:45.303121+0000","last_undegraded":"2024-01-18T14:15:45.303121+0000","last_fullsized":"2024-01-18T14:15:45.303121+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-18T14:15:42.252878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-18T14:15:42.252878+0000","last_clean_scrub_stamp":"2024-01-18T14:15:42.252878+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":614783,"data_compressed":29910,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26988,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344852,"statfs":{"total":96636764160,"available":96609128448,"internally_reserved":0,"allocated":155648,"data_stored":611890,"data_compressed":29910,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-18T14:15:48.841 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-18T14:15:48.953 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:48.949+0000 7fead3d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:48.955 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:48.954+0000 7fead3d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:49.142 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:49.143 INFO:teuthology.orchestra.run.smithi049.stderr:dumped all 2024-01-18T14:15:49.153 INFO:teuthology.orchestra.run.smithi049.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-01-18T14:15:47.245725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80908,"kb_used_data":412,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034612,"statfs":{"total":289910292480,"available":289827442688,"internally_reserved":0,"allocated":421888,"data_stored":1251088,"data_compressed":59820,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.942015"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-18T14:15:45.301574+0000","last_change":"2024-01-18T14:15:44.385487+0000","last_active":"2024-01-18T14:15:45.301574+0000","last_peered":"2024-01-18T14:15:45.301574+0000","last_clean":"2024-01-18T14:15:45.301574+0000","last_became_active":"2024-01-18T14:15:44.383687+0000","last_became_peered":"2024-01-18T14:15:44.383687+0000","last_unstale":"2024-01-18T14:15:45.301574+0000","last_undegraded":"2024-01-18T14:15:45.301574+0000","last_fullsized":"2024-01-18T14:15:45.301574+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_clean_scrub_stamp":"2024-01-18T14:15:43.255804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-18T14:15:45.301027+0000","last_change":"2024-01-18T14:15:44.369074+0000","last_active":"2024-01-18T14:15:45.301027+0000","last_peered":"2024-01-18T14:15:45.301027+0000","last_clean":"2024-01-18T14:15:45.301027+0000","last_became_active":"2024-01-18T14:15:44.364525+0000","last_became_peered":"2024-01-18T14:15:44.364525+0000","last_unstale":"2024-01-18T14:15:45.301027+0000","last_undegraded":"2024-01-18T14:15:45.301027+0000","last_fullsized":"2024-01-18T14:15:45.301027+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_clean_scrub_stamp":"2024-01-18T14:15:43.255804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-18T14:15:45.298913+0000","last_change":"2024-01-18T14:15:44.366299+0000","last_active":"2024-01-18T14:15:45.298913+0000","last_peered":"2024-01-18T14:15:45.298913+0000","last_clean":"2024-01-18T14:15:45.298913+0000","last_became_active":"2024-01-18T14:15:44.361628+0000","last_became_peered":"2024-01-18T14:15:44.361628+0000","last_unstale":"2024-01-18T14:15:45.298913+0000","last_undegraded":"2024-01-18T14:15:45.298913+0000","last_fullsized":"2024-01-18T14:15:45.298913+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_clean_scrub_stamp":"2024-01-18T14:15:43.255804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-18T14:15:45.288193+0000","last_change":"2024-01-18T14:15:44.389001+0000","last_active":"2024-01-18T14:15:45.288193+0000","last_peered":"2024-01-18T14:15:45.288193+0000","last_clean":"2024-01-18T14:15:45.288193+0000","last_became_active":"2024-01-18T14:15:44.384420+0000","last_became_peered":"2024-01-18T14:15:44.384420+0000","last_unstale":"2024-01-18T14:15:45.288193+0000","last_undegraded":"2024-01-18T14:15:45.288193+0000","last_fullsized":"2024-01-18T14:15:45.288193+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_clean_scrub_stamp":"2024-01-18T14:15:43.255804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-18T14:15:45.303392+0000","last_change":"2024-01-18T14:15:44.355497+0000","last_active":"2024-01-18T14:15:45.303392+0000","last_peered":"2024-01-18T14:15:45.303392+0000","last_clean":"2024-01-18T14:15:45.303392+0000","last_became_active":"2024-01-18T14:15:44.353211+0000","last_became_peered":"2024-01-18T14:15:44.353211+0000","last_unstale":"2024-01-18T14:15:45.303392+0000","last_undegraded":"2024-01-18T14:15:45.303392+0000","last_fullsized":"2024-01-18T14:15:45.303392+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_clean_scrub_stamp":"2024-01-18T14:15:43.255804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-18T14:15:45.272827+0000","last_change":"2024-01-18T14:15:44.358997+0000","last_active":"2024-01-18T14:15:45.272827+0000","last_peered":"2024-01-18T14:15:45.272827+0000","last_clean":"2024-01-18T14:15:45.272827+0000","last_became_active":"2024-01-18T14:15:44.356812+0000","last_became_peered":"2024-01-18T14:15:44.356812+0000","last_unstale":"2024-01-18T14:15:45.272827+0000","last_undegraded":"2024-01-18T14:15:45.272827+0000","last_fullsized":"2024-01-18T14:15:45.272827+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_clean_scrub_stamp":"2024-01-18T14:15:43.255804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-18T14:15:45.274052+0000","last_change":"2024-01-18T14:15:44.347366+0000","last_active":"2024-01-18T14:15:45.274052+0000","last_peered":"2024-01-18T14:15:45.274052+0000","last_clean":"2024-01-18T14:15:45.274052+0000","last_became_active":"2024-01-18T14:15:44.344810+0000","last_became_peered":"2024-01-18T14:15:44.344810+0000","last_unstale":"2024-01-18T14:15:45.274052+0000","last_undegraded":"2024-01-18T14:15:45.274052+0000","last_fullsized":"2024-01-18T14:15:45.274052+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_clean_scrub_stamp":"2024-01-18T14:15:43.255804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-18T14:15:45.292046+0000","last_change":"2024-01-18T14:15:44.333666+0000","last_active":"2024-01-18T14:15:45.292046+0000","last_peered":"2024-01-18T14:15:45.292046+0000","last_clean":"2024-01-18T14:15:45.292046+0000","last_became_active":"2024-01-18T14:15:44.331441+0000","last_became_peered":"2024-01-18T14:15:44.331441+0000","last_unstale":"2024-01-18T14:15:45.292046+0000","last_undegraded":"2024-01-18T14:15:45.292046+0000","last_fullsized":"2024-01-18T14:15:45.292046+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-18T14:15:43.255804+0000","last_clean_scrub_stamp":"2024-01-18T14:15:43.255804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-18T14:15:45.303121+0000","last_change":"2024-01-18T14:15:43.329382+0000","last_active":"2024-01-18T14:15:45.303121+0000","last_peered":"2024-01-18T14:15:45.303121+0000","last_clean":"2024-01-18T14:15:45.303121+0000","last_became_active":"2024-01-18T14:15:43.321220+0000","last_became_peered":"2024-01-18T14:15:43.321220+0000","last_unstale":"2024-01-18T14:15:45.303121+0000","last_undegraded":"2024-01-18T14:15:45.303121+0000","last_fullsized":"2024-01-18T14:15:45.303121+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-18T14:15:42.252878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-18T14:15:42.252878+0000","last_clean_scrub_stamp":"2024-01-18T14:15:42.252878+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":614783,"data_compressed":29910,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26988,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344852,"statfs":{"total":96636764160,"available":96609128448,"internally_reserved":0,"allocated":155648,"data_stored":611890,"data_compressed":29910,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-18T14:15:49.155 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-18T14:15:49.155 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-18T14:15:49.155 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-18T14:15:49.155 DEBUG:teuthology.orchestra.run.smithi049:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-18T14:15:49.267 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:49.264+0000 7ff88fc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:49.269 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:15:49.268+0000 7ff88fc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:15:49.495 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:49.495 INFO:teuthology.orchestra.run.smithi049.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-18T14:15:49.507 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-18T14:15:49.507 INFO:teuthology.run_tasks:Running task cbt... 2024-01-18T14:15:49.521 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi049.front.sepia.ceph.com', 'osds': ['ubuntu@smithi049.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi049.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi049.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi049.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randread'], 'norandommap': True, 'op_size': [4096], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2], 'cmd_path': '/home/ubuntu/cephtest/fio/fio'}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-01-18T14:15:49.522 DEBUG:teuthology.orchestra.run.smithi049:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-01-18T14:15:49.539 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-01-18T14:15:49.539 DEBUG:teuthology.orchestra.run.smithi049:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-01-18T14:15:49.593 DEBUG:teuthology.orchestra.run.smithi049:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-01-18T14:15:49.648 INFO:teuthology.orchestra.run.smithi049.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-01-18T14:15:50.074 INFO:tasks.cbt:Installing collectl 2024-01-18T14:15:50.074 DEBUG:teuthology.orchestra.run.smithi049:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-01-18T14:15:50.180 INFO:teuthology.orchestra.run.smithi049.stderr:--2024-01-18 14:15:50-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-01-18T14:15:50.335 INFO:teuthology.orchestra.run.smithi049.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::6812:256f, ... 2024-01-18T14:15:50.344 INFO:teuthology.orchestra.run.smithi049.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected. 2024-01-18T14:15:50.599 INFO:teuthology.orchestra.run.smithi049.stderr:HTTP request sent, awaiting response... 302 Found 2024-01-18T14:15:50.599 INFO:teuthology.orchestra.run.smithi049.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlqTKW15xiMcwK6pbGuXARRxj1o5uRc6lhcbDx_bchqMARxID1L5l7czQqJwNWz6LzRFynZ28GCTX4OKOhq9IeVnTiIA%3D%3D&use_mirror=netactuate&r= [following] 2024-01-18T14:15:50.599 INFO:teuthology.orchestra.run.smithi049.stderr:--2024-01-18 14:15:50-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlqTKW15xiMcwK6pbGuXARRxj1o5uRc6lhcbDx_bchqMARxID1L5l7czQqJwNWz6LzRFynZ28GCTX4OKOhq9IeVnTiIA%3D%3D&use_mirror=netactuate&r= 2024-01-18T14:15:50.609 INFO:teuthology.orchestra.run.smithi049.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-01-18T14:15:50.680 INFO:teuthology.orchestra.run.smithi049.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-01-18T14:15:50.918 INFO:teuthology.orchestra.run.smithi049.stderr:HTTP request sent, awaiting response... 302 Found 2024-01-18T14:15:50.918 INFO:teuthology.orchestra.run.smithi049.stderr:Location: https://netactuate.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2024-01-18T14:15:50.918 INFO:teuthology.orchestra.run.smithi049.stderr:--2024-01-18 14:15:50-- https://netactuate.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2024-01-18T14:15:50.929 INFO:teuthology.orchestra.run.smithi049.stderr:Resolving netactuate.dl.sourceforge.net (netactuate.dl.sourceforge.net)... 104.225.3.66 2024-01-18T14:15:50.942 INFO:teuthology.orchestra.run.smithi049.stderr:Connecting to netactuate.dl.sourceforge.net (netactuate.dl.sourceforge.net)|104.225.3.66|:443... connected. 2024-01-18T14:15:51.079 INFO:teuthology.orchestra.run.smithi049.stderr:HTTP request sent, awaiting response... 200 OK 2024-01-18T14:15:51.079 INFO:teuthology.orchestra.run.smithi049.stderr:Length: 620693 (606K) [application/x-gzip] 2024-01-18T14:15:51.080 INFO:teuthology.orchestra.run.smithi049.stderr:Saving to: ‘collectl.tar.gz’ 2024-01-18T14:15:51.080 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-01-18T14:15:51.092 INFO:teuthology.orchestra.run.smithi049.stderr: 0K .......... .......... .......... .......... .......... 8% 3.84M 0s 2024-01-18T14:15:51.105 INFO:teuthology.orchestra.run.smithi049.stderr: 50K .......... .......... .......... .......... .......... 16% 3.89M 0s 2024-01-18T14:15:51.105 INFO:teuthology.orchestra.run.smithi049.stderr: 100K .......... .......... .......... .......... .......... 24% 207M 0s 2024-01-18T14:15:51.119 INFO:teuthology.orchestra.run.smithi049.stderr: 150K .......... .......... .......... .......... .......... 32% 3.95M 0s 2024-01-18T14:15:51.119 INFO:teuthology.orchestra.run.smithi049.stderr: 200K .......... .......... .......... .......... .......... 41% 144M 0s 2024-01-18T14:15:51.119 INFO:teuthology.orchestra.run.smithi049.stderr: 250K .......... .......... .......... .......... .......... 49% 348M 0s 2024-01-18T14:15:51.119 INFO:teuthology.orchestra.run.smithi049.stderr: 300K .......... .......... .......... .......... .......... 57% 329M 0s 2024-01-18T14:15:51.133 INFO:teuthology.orchestra.run.smithi049.stderr: 350K .......... .......... .......... .......... .......... 65% 4.04M 0s 2024-01-18T14:15:51.133 INFO:teuthology.orchestra.run.smithi049.stderr: 400K .......... .......... .......... .......... .......... 74% 321M 0s 2024-01-18T14:15:51.133 INFO:teuthology.orchestra.run.smithi049.stderr: 450K .......... .......... .......... .......... .......... 82% 346M 0s 2024-01-18T14:15:51.133 INFO:teuthology.orchestra.run.smithi049.stderr: 500K .......... .......... .......... .......... .......... 90% 344M 0s 2024-01-18T14:15:51.133 INFO:teuthology.orchestra.run.smithi049.stderr: 550K .......... .......... .......... .......... .......... 98% 321M 0s 2024-01-18T14:15:51.133 INFO:teuthology.orchestra.run.smithi049.stderr: 600K ...... 100% 11.4T=0.05s 2024-01-18T14:15:51.133 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-01-18T14:15:51.133 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18 14:15:51 (11.6 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-01-18T14:15:51.133 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-01-18T14:15:51.166 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/ 2024-01-18T14:15:51.166 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/collectl.conf 2024-01-18T14:15:51.166 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/ 2024-01-18T14:15:51.166 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/SlowProc.html 2024-01-18T14:15:51.166 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-01-18T14:15:51.166 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/WhySummary.html 2024-01-18T14:15:51.166 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Socket.html 2024-01-18T14:15:51.166 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/colmux.html 2024-01-18T14:15:51.166 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Tutorial.html 2024-01-18T14:15:51.166 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Gexpr.html 2024-01-18T14:15:51.166 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Import.html 2024-01-18T14:15:51.166 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/style.css 2024-01-18T14:15:51.166 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-01-18T14:15:51.167 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-01-18T14:15:51.167 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/FileNaming.html 2024-01-18T14:15:51.167 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/TheMath.html 2024-01-18T14:15:51.167 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Environmental.html 2024-01-18T14:15:51.167 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Process.html 2024-01-18T14:15:51.167 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-01-18T14:15:51.167 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/HiResTime.html 2024-01-18T14:15:51.167 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-01-18T14:15:51.167 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/slub.jpg 2024-01-18T14:15:51.167 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Lustre.html 2024-01-18T14:15:51.167 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Documentation.html 2024-01-18T14:15:51.167 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Inodes.html 2024-01-18T14:15:51.167 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Graphite.html 2024-01-18T14:15:51.167 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/InputFiles.html 2024-01-18T14:15:51.167 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Export.html 2024-01-18T14:15:51.168 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Interrupts.html 2024-01-18T14:15:51.168 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Startup.html 2024-01-18T14:15:51.168 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Performance.html 2024-01-18T14:15:51.168 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Messages.html 2024-01-18T14:15:51.168 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Logging.html 2024-01-18T14:15:51.168 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Data-detail.html 2024-01-18T14:15:51.168 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/CPUs.html 2024-01-18T14:15:51.168 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Matrix.html 2024-01-18T14:15:51.168 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Data-brief.html 2024-01-18T14:15:51.168 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-01-18T14:15:51.168 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Misc.html 2024-01-18T14:15:51.168 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Examples.html 2024-01-18T14:15:51.168 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/OpenStack.html 2024-01-18T14:15:51.169 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-01-18T14:15:51.169 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-01-18T14:15:51.169 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-01-18T14:15:51.169 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-01-18T14:15:51.169 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Data.html 2024-01-18T14:15:51.169 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Exceptions.html 2024-01-18T14:15:51.169 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-01-18T14:15:51.169 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-01-18T14:15:51.169 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-01-18T14:15:51.169 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-01-18T14:15:51.169 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Playback.html 2024-01-18T14:15:51.169 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Hello.html 2024-01-18T14:15:51.169 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Sockets.html 2024-01-18T14:15:51.169 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Network.html 2024-01-18T14:15:51.169 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Infiniband.html 2024-01-18T14:15:51.169 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Memory.html 2024-01-18T14:15:51.170 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-01-18T14:15:51.170 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/Disks.html 2024-01-18T14:15:51.170 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-01-18T14:15:51.170 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/service/ 2024-01-18T14:15:51.170 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/service/collectl.service 2024-01-18T14:15:51.170 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/initd/ 2024-01-18T14:15:51.170 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/initd/collectl-suse 2024-01-18T14:15:51.170 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/initd/collectl-debian 2024-01-18T14:15:51.170 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/initd/collectl-generic 2024-01-18T14:15:51.170 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/initd/collectl 2024-01-18T14:15:51.170 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/misc.ph 2024-01-18T14:15:51.170 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/gexpr.ph 2024-01-18T14:15:51.170 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/statsd.ph 2024-01-18T14:15:51.170 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/envrules.std 2024-01-18T14:15:51.170 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/hello.ph 2024-01-18T14:15:51.171 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/vmsum.ph 2024-01-18T14:15:51.171 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/proctree.ph 2024-01-18T14:15:51.171 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/README-WINDOWS 2024-01-18T14:15:51.171 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/formatit.ph 2024-01-18T14:15:51.171 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/colmux 2024-01-18T14:15:51.173 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/INSTALL 2024-01-18T14:15:51.173 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/graphite.ph 2024-01-18T14:15:51.173 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/RELEASE-collectl 2024-01-18T14:15:51.173 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/vmstat.ph 2024-01-18T14:15:51.173 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/README 2024-01-18T14:15:51.173 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/man1/ 2024-01-18T14:15:51.173 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/man1/collectl.1 2024-01-18T14:15:51.173 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/man1/colmux.1 2024-01-18T14:15:51.173 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/lexpr.ph 2024-01-18T14:15:51.173 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/client.pl 2024-01-18T14:15:51.173 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/collectl 2024-01-18T14:15:51.173 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/COPYING 2024-01-18T14:15:51.173 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/vnet.ph 2024-01-18T14:15:51.173 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/GPL 2024-01-18T14:15:51.174 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/UNINSTALL 2024-01-18T14:15:51.174 INFO:teuthology.orchestra.run.smithi049.stdout:collectl-4.3.1/ARTISTIC 2024-01-18T14:15:51.229 INFO:teuthology.orchestra.run.smithi049.stderr:./INSTALL: line 90: chkconfig: command not found 2024-01-18T14:15:51.231 DEBUG:teuthology.orchestra.run.smithi049:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-01-18T14:15:51.815 INFO:teuthology.orchestra.run.smithi049.stdout:Last metadata expiration check: 0:03:28 ago on Thu 18 Jan 2024 02:12:23 PM UTC. 2024-01-18T14:15:51.920 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-01-18T14:15:51.922 INFO:teuthology.orchestra.run.smithi049.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-01-18T14:15:51.981 INFO:teuthology.orchestra.run.smithi049.stdout:Dependencies resolved. 2024-01-18T14:15:51.981 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:15:51.981 INFO:teuthology.orchestra.run.smithi049.stdout: Package Arch Version Repository Size 2024-01-18T14:15:51.981 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:15:51.982 INFO:teuthology.orchestra.run.smithi049.stdout:Installing: 2024-01-18T14:15:51.982 INFO:teuthology.orchestra.run.smithi049.stdout: librbd-devel x86_64 2:19.0.0-749.g45e21eee.el9 ceph 23 k 2024-01-18T14:15:51.982 INFO:teuthology.orchestra.run.smithi049.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-01-18T14:15:51.982 INFO:teuthology.orchestra.run.smithi049.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-01-18T14:15:51.982 INFO:teuthology.orchestra.run.smithi049.stdout: perf x86_64 5.14.0-408.el9 CentOS-AppStream 8.0 M 2024-01-18T14:15:51.982 INFO:teuthology.orchestra.run.smithi049.stdout:Installing dependencies: 2024-01-18T14:15:51.982 INFO:teuthology.orchestra.run.smithi049.stdout: libradospp-devel x86_64 2:19.0.0-749.g45e21eee.el9 ceph 30 k 2024-01-18T14:15:51.982 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:51.982 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction Summary 2024-01-18T14:15:51.982 INFO:teuthology.orchestra.run.smithi049.stdout:================================================================================ 2024-01-18T14:15:51.982 INFO:teuthology.orchestra.run.smithi049.stdout:Install 5 Packages 2024-01-18T14:15:51.982 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:51.983 INFO:teuthology.orchestra.run.smithi049.stdout:Total download size: 8.1 M 2024-01-18T14:15:51.983 INFO:teuthology.orchestra.run.smithi049.stdout:Installed size: 11 M 2024-01-18T14:15:51.983 INFO:teuthology.orchestra.run.smithi049.stdout:Downloading Packages: 2024-01-18T14:15:52.444 INFO:teuthology.orchestra.run.smithi049.stdout:(1/5): perf-5.14.0-408.el9.x86_64.rpm 25 MB/s | 8.0 MB 00:00 2024-01-18T14:15:52.603 INFO:teuthology.orchestra.run.smithi049.stdout:(2/5): libradospp-devel-19.0.0-749.g45e21eee.el 64 kB/s | 30 kB 00:00 2024-01-18T14:15:52.628 INFO:teuthology.orchestra.run.smithi049.stdout:(3/5): librbd-devel-19.0.0-749.g45e21eee.el9.x8 47 kB/s | 23 kB 00:00 2024-01-18T14:15:52.654 INFO:teuthology.orchestra.run.smithi049.stdout:(4/5): pdsh-2.34-7.el9.x86_64.rpm 568 kB/s | 118 kB 00:00 2024-01-18T14:15:52.670 INFO:teuthology.orchestra.run.smithi049.stdout:(5/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 197 kB/s | 13 kB 00:00 2024-01-18T14:15:52.673 INFO:teuthology.orchestra.run.smithi049.stdout:-------------------------------------------------------------------------------- 2024-01-18T14:15:52.673 INFO:teuthology.orchestra.run.smithi049.stdout:Total 12 MB/s | 8.1 MB 00:00 2024-01-18T14:15:52.673 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction check 2024-01-18T14:15:52.695 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction check succeeded. 2024-01-18T14:15:52.696 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction test 2024-01-18T14:15:52.866 INFO:teuthology.orchestra.run.smithi049.stdout:Transaction test succeeded. 2024-01-18T14:15:52.869 INFO:teuthology.orchestra.run.smithi049.stdout:Running transaction 2024-01-18T14:15:53.243 INFO:teuthology.orchestra.run.smithi049.stdout: Preparing : 1/1 2024-01-18T14:15:53.307 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-01-18T14:15:53.336 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-01-18T14:15:53.359 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : libradospp-devel-2:19.0.0-749.g45e21eee.el9.x86_64 3/5 2024-01-18T14:15:53.695 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : librbd-devel-2:19.0.0-749.g45e21eee.el9.x86_64 4/5 2024-01-18T14:15:53.834 INFO:teuthology.orchestra.run.smithi049.stdout: Installing : perf-5.14.0-408.el9.x86_64 5/5 2024-01-18T14:15:54.754 INFO:teuthology.orchestra.run.smithi049.stdout: Running scriptlet: perf-5.14.0-408.el9.x86_64 5/5 2024-01-18T14:15:54.754 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : libradospp-devel-2:19.0.0-749.g45e21eee.el9.x86_64 1/5 2024-01-18T14:15:54.755 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : librbd-devel-2:19.0.0-749.g45e21eee.el9.x86_64 2/5 2024-01-18T14:15:54.755 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : perf-5.14.0-408.el9.x86_64 3/5 2024-01-18T14:15:54.755 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-01-18T14:15:55.120 INFO:teuthology.orchestra.run.smithi049.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-01-18T14:15:55.120 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:55.121 INFO:teuthology.orchestra.run.smithi049.stdout:Installed: 2024-01-18T14:15:55.121 INFO:teuthology.orchestra.run.smithi049.stdout: libradospp-devel-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:15:55.121 INFO:teuthology.orchestra.run.smithi049.stdout: librbd-devel-2:19.0.0-749.g45e21eee.el9.x86_64 2024-01-18T14:15:55.121 INFO:teuthology.orchestra.run.smithi049.stdout: pdsh-2.34-7.el9.x86_64 2024-01-18T14:15:55.121 INFO:teuthology.orchestra.run.smithi049.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-01-18T14:15:55.121 INFO:teuthology.orchestra.run.smithi049.stdout: perf-5.14.0-408.el9.x86_64 2024-01-18T14:15:55.121 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:55.121 INFO:teuthology.orchestra.run.smithi049.stdout:Complete! 2024-01-18T14:15:55.477 INFO:tasks.cbt:benchmark: librbdfio 2024-01-18T14:15:55.477 DEBUG:teuthology.orchestra.run.smithi049:> git clone -b master https://github.com/axboe/fio.git /home/ubuntu/cephtest/fio 2024-01-18T14:15:55.492 INFO:teuthology.orchestra.run.smithi049.stderr:Cloning into '/home/ubuntu/cephtest/fio'... 2024-01-18T14:15:57.708 DEBUG:teuthology.orchestra.run.smithi049:> cd /home/ubuntu/cephtest/fio && ./configure && make 2024-01-18T14:15:58.243 INFO:teuthology.orchestra.run.smithi049.stdout:Operating system Linux 2024-01-18T14:15:58.243 INFO:teuthology.orchestra.run.smithi049.stdout:CPU x86_64 2024-01-18T14:15:58.244 INFO:teuthology.orchestra.run.smithi049.stdout:Big endian no 2024-01-18T14:15:58.244 INFO:teuthology.orchestra.run.smithi049.stdout:Compiler gcc 2024-01-18T14:15:58.244 INFO:teuthology.orchestra.run.smithi049.stdout:Cross compile no 2024-01-18T14:15:58.244 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:15:58.244 INFO:teuthology.orchestra.run.smithi049.stdout:Static build no 2024-01-18T14:15:58.359 INFO:teuthology.orchestra.run.smithi049.stdout:Wordsize 64 2024-01-18T14:15:58.367 INFO:teuthology.orchestra.run.smithi049.stdout:zlib no 2024-01-18T14:15:58.420 INFO:teuthology.orchestra.run.smithi049.stdout:fcntl(F_FULLFSYNC) no 2024-01-18T14:15:58.435 INFO:teuthology.orchestra.run.smithi049.stdout:Linux AIO support no 2024-01-18T14:15:58.435 INFO:teuthology.orchestra.run.smithi049.stdout:Linux AIO support rw flags no 2024-01-18T14:15:58.504 INFO:teuthology.orchestra.run.smithi049.stdout:POSIX AIO support yes 2024-01-18T14:15:58.504 INFO:teuthology.orchestra.run.smithi049.stdout:POSIX AIO support needs -lrt no 2024-01-18T14:15:58.536 INFO:teuthology.orchestra.run.smithi049.stdout:POSIX AIO fsync yes 2024-01-18T14:15:58.566 INFO:teuthology.orchestra.run.smithi049.stdout:POSIX pshared support yes 2024-01-18T14:15:58.619 INFO:teuthology.orchestra.run.smithi049.stdout:pthread_condattr_setclock() yes 2024-01-18T14:15:58.658 INFO:teuthology.orchestra.run.smithi049.stdout:pthread_sigmask() yes 2024-01-18T14:15:58.696 INFO:teuthology.orchestra.run.smithi049.stdout:pthread_getaffinity_np() yes 2024-01-18T14:15:58.706 INFO:teuthology.orchestra.run.smithi049.stdout:Solaris AIO support no 2024-01-18T14:15:58.751 INFO:teuthology.orchestra.run.smithi049.stdout:__sync_fetch_and_add yes 2024-01-18T14:15:58.790 INFO:teuthology.orchestra.run.smithi049.stdout:__sync_synchronize yes 2024-01-18T14:15:58.820 INFO:teuthology.orchestra.run.smithi049.stdout:__sync_val_compare_and_swap yes 2024-01-18T14:15:58.827 INFO:teuthology.orchestra.run.smithi049.stdout:libverbs no 2024-01-18T14:15:58.851 INFO:teuthology.orchestra.run.smithi049.stdout:rdmacm no 2024-01-18T14:15:58.858 INFO:teuthology.orchestra.run.smithi049.stdout:librpma no 2024-01-18T14:15:58.868 INFO:teuthology.orchestra.run.smithi049.stdout:libprotobuf_c no 2024-01-18T14:15:58.899 INFO:teuthology.orchestra.run.smithi049.stdout:asprintf() yes 2024-01-18T14:15:58.931 INFO:teuthology.orchestra.run.smithi049.stdout:vasprintf() yes 2024-01-18T14:15:58.963 INFO:teuthology.orchestra.run.smithi049.stdout:Linux fallocate yes 2024-01-18T14:15:58.995 INFO:teuthology.orchestra.run.smithi049.stdout:POSIX fadvise yes 2024-01-18T14:15:59.027 INFO:teuthology.orchestra.run.smithi049.stdout:POSIX fallocate yes 2024-01-18T14:15:59.057 INFO:teuthology.orchestra.run.smithi049.stdout:sched_setaffinity(3 arg) yes 2024-01-18T14:15:59.057 INFO:teuthology.orchestra.run.smithi049.stdout:sched_setaffinity(2 arg) no 2024-01-18T14:15:59.089 INFO:teuthology.orchestra.run.smithi049.stdout:clock_gettime yes 2024-01-18T14:15:59.123 INFO:teuthology.orchestra.run.smithi049.stdout:CLOCK_MONOTONIC yes 2024-01-18T14:15:59.171 INFO:teuthology.orchestra.run.smithi049.stdout:clockid_t yes 2024-01-18T14:15:59.209 INFO:teuthology.orchestra.run.smithi049.stdout:gettimeofday yes 2024-01-18T14:15:59.243 INFO:teuthology.orchestra.run.smithi049.stdout:fdatasync yes 2024-01-18T14:15:59.274 INFO:teuthology.orchestra.run.smithi049.stdout:pipe() yes 2024-01-18T14:15:59.306 INFO:teuthology.orchestra.run.smithi049.stdout:pipe2() yes 2024-01-18T14:15:59.337 INFO:teuthology.orchestra.run.smithi049.stdout:pread() yes 2024-01-18T14:15:59.423 INFO:teuthology.orchestra.run.smithi049.stdout:sync_file_range yes 2024-01-18T14:15:59.430 INFO:teuthology.orchestra.run.smithi049.stdout:ASharedMemory_create no 2024-01-18T14:15:59.442 INFO:teuthology.orchestra.run.smithi049.stdout:EXT4 move extent yes 2024-01-18T14:15:59.475 INFO:teuthology.orchestra.run.smithi049.stdout:Linux splice(2) yes 2024-01-18T14:15:59.482 INFO:teuthology.orchestra.run.smithi049.stdout:libnuma no 2024-01-18T14:15:59.514 INFO:teuthology.orchestra.run.smithi049.stdout:strsep yes 2024-01-18T14:15:59.544 INFO:teuthology.orchestra.run.smithi049.stdout:strcasestr yes 2024-01-18T14:15:59.558 INFO:teuthology.orchestra.run.smithi049.stdout:strlcat no 2024-01-18T14:15:59.604 INFO:teuthology.orchestra.run.smithi049.stdout:getopt_long_only() yes 2024-01-18T14:15:59.669 INFO:teuthology.orchestra.run.smithi049.stdout:inet_aton yes 2024-01-18T14:15:59.700 INFO:teuthology.orchestra.run.smithi049.stdout:socklen_t yes 2024-01-18T14:15:59.733 INFO:teuthology.orchestra.run.smithi049.stdout:__thread yes 2024-01-18T14:15:59.774 INFO:teuthology.orchestra.run.smithi049.stdout:RUSAGE_THREAD yes 2024-01-18T14:15:59.804 INFO:teuthology.orchestra.run.smithi049.stdout:SCHED_IDLE yes 2024-01-18T14:15:59.839 INFO:teuthology.orchestra.run.smithi049.stdout:TCP_NODELAY yes 2024-01-18T14:15:59.874 INFO:teuthology.orchestra.run.smithi049.stdout:Net engine window_size yes 2024-01-18T14:15:59.910 INFO:teuthology.orchestra.run.smithi049.stdout:TCP_MAXSEG yes 2024-01-18T14:15:59.940 INFO:teuthology.orchestra.run.smithi049.stdout:RLIMIT_MEMLOCK yes 2024-01-18T14:15:59.974 INFO:teuthology.orchestra.run.smithi049.stdout:pwritev/preadv yes 2024-01-18T14:16:00.008 INFO:teuthology.orchestra.run.smithi049.stdout:pwritev2/preadv2 yes 2024-01-18T14:16:00.047 INFO:teuthology.orchestra.run.smithi049.stdout:IPv6 helpers yes 2024-01-18T14:16:00.062 INFO:teuthology.orchestra.run.smithi049.stdout:http engine no 2024-01-18T14:16:00.159 INFO:teuthology.orchestra.run.smithi049.stdout:Rados engine yes 2024-01-18T14:16:00.266 INFO:teuthology.orchestra.run.smithi049.stdout:Rados Block Device engine yes 2024-01-18T14:16:00.390 INFO:teuthology.orchestra.run.smithi049.stdout:rbd_poll yes 2024-01-18T14:16:00.498 INFO:teuthology.orchestra.run.smithi049.stdout:rbd_invalidate_cache yes 2024-01-18T14:16:00.530 INFO:teuthology.orchestra.run.smithi049.stdout:setvbuf yes 2024-01-18T14:16:00.537 INFO:teuthology.orchestra.run.smithi049.stdout:Gluster API engine no 2024-01-18T14:16:00.553 INFO:teuthology.orchestra.run.smithi049.stdout:s390_z196_facilities no 2024-01-18T14:16:00.553 INFO:teuthology.orchestra.run.smithi049.stdout:HDFS engine no 2024-01-18T14:16:00.595 INFO:teuthology.orchestra.run.smithi049.stdout:MTD yes 2024-01-18T14:16:00.603 INFO:teuthology.orchestra.run.smithi049.stdout:libpmem no 2024-01-18T14:16:00.603 INFO:teuthology.orchestra.run.smithi049.stdout:libpmem1_5 no 2024-01-18T14:16:00.610 INFO:teuthology.orchestra.run.smithi049.stdout:libpmem2 no 2024-01-18T14:16:00.610 INFO:teuthology.orchestra.run.smithi049.stdout:PMDK dev-dax engine no 2024-01-18T14:16:00.611 INFO:teuthology.orchestra.run.smithi049.stdout:PMDK libpmem engine no 2024-01-18T14:16:00.618 INFO:teuthology.orchestra.run.smithi049.stdout:DDN's Infinite Memory Engine no 2024-01-18T14:16:00.618 INFO:teuthology.orchestra.run.smithi049.stdout:iscsi engine no 2024-01-18T14:16:00.618 INFO:teuthology.orchestra.run.smithi049.stdout:NBD engine no 2024-01-18T14:16:00.627 INFO:teuthology.orchestra.run.smithi049.stdout:DAOS File System (dfs) Engine no 2024-01-18T14:16:00.636 INFO:teuthology.orchestra.run.smithi049.stdout:NFS engine no 2024-01-18T14:16:00.636 INFO:teuthology.orchestra.run.smithi049.stdout:lex/yacc for arithmetic no 2024-01-18T14:16:00.670 INFO:teuthology.orchestra.run.smithi049.stdout:getmntent yes 2024-01-18T14:16:00.691 INFO:teuthology.orchestra.run.smithi049.stdout:getmntinfo no 2024-01-18T14:16:00.753 INFO:teuthology.orchestra.run.smithi049.stdout:Static Assert yes 2024-01-18T14:16:00.782 INFO:teuthology.orchestra.run.smithi049.stdout:bool yes 2024-01-18T14:16:00.816 INFO:teuthology.orchestra.run.smithi049.stdout:strndup yes 2024-01-18T14:16:00.823 INFO:teuthology.orchestra.run.smithi049.stdout:Valgrind headers no 2024-01-18T14:16:00.874 INFO:teuthology.orchestra.run.smithi049.stdout:Zoned block device support yes 2024-01-18T14:16:00.902 INFO:teuthology.orchestra.run.smithi049.stdout:Zoned block device capacity yes 2024-01-18T14:16:00.909 INFO:teuthology.orchestra.run.smithi049.stdout:libzbc engine no 2024-01-18T14:16:00.951 INFO:teuthology.orchestra.run.smithi049.stdout:NVMe uring command support yes 2024-01-18T14:16:00.966 INFO:teuthology.orchestra.run.smithi049.stdout:xnvme engine no 2024-01-18T14:16:00.973 INFO:teuthology.orchestra.run.smithi049.stdout:isal no 2024-01-18T14:16:00.988 INFO:teuthology.orchestra.run.smithi049.stdout:libblkio engine no 2024-01-18T14:16:00.988 INFO:teuthology.orchestra.run.smithi049.stdout:march_armv8_a_crc_crypto no 2024-01-18T14:16:00.988 INFO:teuthology.orchestra.run.smithi049.stdout:cuda no 2024-01-18T14:16:00.989 INFO:teuthology.orchestra.run.smithi049.stdout:libcufile no 2024-01-18T14:16:01.017 INFO:teuthology.orchestra.run.smithi049.stdout:Build march=native yes 2024-01-18T14:16:01.025 INFO:teuthology.orchestra.run.smithi049.stdout:CUnit no 2024-01-18T14:16:01.055 INFO:teuthology.orchestra.run.smithi049.stdout:__kernel_rwf_t yes 2024-01-18T14:16:01.084 INFO:teuthology.orchestra.run.smithi049.stdout:-Wimplicit-fallthrough=2 yes 2024-01-18T14:16:01.115 INFO:teuthology.orchestra.run.smithi049.stdout:-Wno-stringop-truncation yes 2024-01-18T14:16:01.146 INFO:teuthology.orchestra.run.smithi049.stdout:MADV_HUGEPAGE yes 2024-01-18T14:16:01.177 INFO:teuthology.orchestra.run.smithi049.stdout:gettid yes 2024-01-18T14:16:01.212 INFO:teuthology.orchestra.run.smithi049.stdout:statx(2)/libc yes 2024-01-18T14:16:01.260 INFO:teuthology.orchestra.run.smithi049.stdout:statx(2)/syscall yes 2024-01-18T14:16:01.263 INFO:teuthology.orchestra.run.smithi049.stdout:Windows PDB generation no 2024-01-18T14:16:01.295 INFO:teuthology.orchestra.run.smithi049.stdout:timerfd_create yes 2024-01-18T14:16:01.297 INFO:teuthology.orchestra.run.smithi049.stdout:Consider installing zlib1g-dev (zlib-devel) as some fio features depend on it. 2024-01-18T14:16:01.297 INFO:teuthology.orchestra.run.smithi049.stdout:Lib-based ioengines dynamic no 2024-01-18T14:16:01.362 INFO:teuthology.orchestra.run.smithi049.stdout:TCMalloc support yes 2024-01-18T14:16:01.363 INFO:teuthology.orchestra.run.smithi049.stdout:seed_buckets 4 2024-01-18T14:16:01.447 INFO:teuthology.orchestra.run.smithi049.stderr:FIO_VERSION = fio-3.36-41-g9eef 2024-01-18T14:16:01.481 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/crc16.o 2024-01-18T14:16:01.570 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/crc32.o 2024-01-18T14:16:01.607 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/crc32c-arm64.o 2024-01-18T14:16:01.687 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/crc32c-intel.o 2024-01-18T14:16:01.732 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/crc32c.o 2024-01-18T14:16:01.771 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/crc64.o 2024-01-18T14:16:01.811 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/crc7.o 2024-01-18T14:16:01.845 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/crct10dif_common.o 2024-01-18T14:16:01.880 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/fnv.o 2024-01-18T14:16:01.921 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/md5.o 2024-01-18T14:16:02.030 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/murmur3.o 2024-01-18T14:16:02.074 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/sha1.o 2024-01-18T14:16:02.262 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/sha256.o 2024-01-18T14:16:02.570 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/sha3.o 2024-01-18T14:16:02.719 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/sha512.o 2024-01-18T14:16:02.830 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/test.o 2024-01-18T14:16:03.001 INFO:teuthology.orchestra.run.smithi049.stdout: CC crc/xxhash.o 2024-01-18T14:16:03.099 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/axmap.o 2024-01-18T14:16:03.226 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/bloom.o 2024-01-18T14:16:03.311 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/flist_sort.o 2024-01-18T14:16:03.384 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/gauss.o 2024-01-18T14:16:03.484 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/getrusage.o 2024-01-18T14:16:03.519 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/hweight.o 2024-01-18T14:16:03.558 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/ieee754.o 2024-01-18T14:16:03.609 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/lfsr.o 2024-01-18T14:16:03.713 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/memalign.o 2024-01-18T14:16:03.758 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/memcpy.o 2024-01-18T14:16:03.908 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/mountcheck.o 2024-01-18T14:16:03.949 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/num2str.o 2024-01-18T14:16:04.016 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/output_buffer.o 2024-01-18T14:16:04.065 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/pattern.o 2024-01-18T14:16:04.215 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/prio_tree.o 2024-01-18T14:16:04.341 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/rand.o 2024-01-18T14:16:04.535 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/rbtree.o 2024-01-18T14:16:04.665 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/strntol.o 2024-01-18T14:16:04.713 INFO:teuthology.orchestra.run.smithi049.stdout: CC lib/zipf.o 2024-01-18T14:16:04.836 INFO:teuthology.orchestra.run.smithi049.stdout: CC gettime.o 2024-01-18T14:16:05.138 INFO:teuthology.orchestra.run.smithi049.stdout: CC ioengines.o 2024-01-18T14:16:05.414 INFO:teuthology.orchestra.run.smithi049.stdout: CC init.o 2024-01-18T14:16:06.198 INFO:teuthology.orchestra.run.smithi049.stdout: CC stat.o 2024-01-18T14:16:08.147 INFO:teuthology.orchestra.run.smithi049.stdout: CC log.o 2024-01-18T14:16:08.276 INFO:teuthology.orchestra.run.smithi049.stdout: CC time.o 2024-01-18T14:16:08.404 INFO:teuthology.orchestra.run.smithi049.stdout: CC filesetup.o 2024-01-18T14:16:09.129 INFO:teuthology.orchestra.run.smithi049.stdout: CC eta.o 2024-01-18T14:16:09.457 INFO:teuthology.orchestra.run.smithi049.stdout: CC verify.o 2024-01-18T14:16:10.156 INFO:teuthology.orchestra.run.smithi049.stdout: CC memory.o 2024-01-18T14:16:10.330 INFO:teuthology.orchestra.run.smithi049.stdout: CC io_u.o 2024-01-18T14:16:11.113 INFO:teuthology.orchestra.run.smithi049.stdout: CC parse.o 2024-01-18T14:16:11.572 INFO:teuthology.orchestra.run.smithi049.stdout: CC fio_sem.o 2024-01-18T14:16:11.681 INFO:teuthology.orchestra.run.smithi049.stdout: CC rwlock.o 2024-01-18T14:16:11.765 INFO:teuthology.orchestra.run.smithi049.stdout: CC pshared.o 2024-01-18T14:16:11.831 INFO:teuthology.orchestra.run.smithi049.stdout: CC options.o 2024-01-18T14:16:12.641 INFO:teuthology.orchestra.run.smithi049.stdout: CC smalloc.o 2024-01-18T14:16:12.836 INFO:teuthology.orchestra.run.smithi049.stdout: CC filehash.o 2024-01-18T14:16:12.991 INFO:teuthology.orchestra.run.smithi049.stdout: CC profile.o 2024-01-18T14:16:13.112 INFO:teuthology.orchestra.run.smithi049.stdout: CC debug.o 2024-01-18T14:16:13.153 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/cpu.o 2024-01-18T14:16:13.298 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/mmap.o 2024-01-18T14:16:13.472 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/sync.o 2024-01-18T14:16:13.706 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/null.o 2024-01-18T14:16:13.824 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/net.o 2024-01-18T14:16:14.250 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/ftruncate.o 2024-01-18T14:16:14.353 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/fileoperations.o 2024-01-18T14:16:14.497 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/exec.o 2024-01-18T14:16:14.652 INFO:teuthology.orchestra.run.smithi049.stdout: CC server.o 2024-01-18T14:16:15.420 INFO:teuthology.orchestra.run.smithi049.stdout: CC client.o 2024-01-18T14:16:15.977 INFO:teuthology.orchestra.run.smithi049.stdout: CC iolog.o 2024-01-18T14:16:16.442 INFO:teuthology.orchestra.run.smithi049.stdout: CC backend.o 2024-01-18T14:16:17.317 INFO:teuthology.orchestra.run.smithi049.stdout: CC libfio.o 2024-01-18T14:16:17.505 INFO:teuthology.orchestra.run.smithi049.stdout: CC flow.o 2024-01-18T14:16:17.621 INFO:teuthology.orchestra.run.smithi049.stdout: CC cconv.o 2024-01-18T14:16:17.984 INFO:teuthology.orchestra.run.smithi049.stdout: CC gettime-thread.o 2024-01-18T14:16:18.088 INFO:teuthology.orchestra.run.smithi049.stdout: CC helpers.o 2024-01-18T14:16:18.152 INFO:teuthology.orchestra.run.smithi049.stdout: CC json.o 2024-01-18T14:16:18.339 INFO:teuthology.orchestra.run.smithi049.stdout: CC idletime.o 2024-01-18T14:16:18.573 INFO:teuthology.orchestra.run.smithi049.stdout: CC td_error.o 2024-01-18T14:16:18.672 INFO:teuthology.orchestra.run.smithi049.stdout: CC profiles/tiobench.o 2024-01-18T14:16:18.771 INFO:teuthology.orchestra.run.smithi049.stdout: CC profiles/act.o 2024-01-18T14:16:18.939 INFO:teuthology.orchestra.run.smithi049.stdout: CC io_u_queue.o 2024-01-18T14:16:18.990 INFO:teuthology.orchestra.run.smithi049.stdout: CC filelock.o 2024-01-18T14:16:19.121 INFO:teuthology.orchestra.run.smithi049.stdout: CC workqueue.o 2024-01-18T14:16:19.299 INFO:teuthology.orchestra.run.smithi049.stdout: CC rate-submit.o 2024-01-18T14:16:19.451 INFO:teuthology.orchestra.run.smithi049.stdout: CC optgroup.o 2024-01-18T14:16:19.498 INFO:teuthology.orchestra.run.smithi049.stdout: CC helper_thread.o 2024-01-18T14:16:19.682 INFO:teuthology.orchestra.run.smithi049.stdout: CC steadystate.o 2024-01-18T14:16:19.924 INFO:teuthology.orchestra.run.smithi049.stdout: CC zone-dist.o 2024-01-18T14:16:20.064 INFO:teuthology.orchestra.run.smithi049.stdout: CC zbd.o 2024-01-18T14:16:20.783 INFO:teuthology.orchestra.run.smithi049.stdout: CC dedupe.o 2024-01-18T14:16:20.929 INFO:teuthology.orchestra.run.smithi049.stdout: CC fdp.o 2024-01-18T14:16:21.074 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/posixaio.o 2024-01-18T14:16:21.213 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/falloc.o 2024-01-18T14:16:21.331 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/e4defrag.o 2024-01-18T14:16:21.470 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/splice.o 2024-01-18T14:16:21.664 INFO:teuthology.orchestra.run.smithi049.stdout: CC oslib/asprintf.o 2024-01-18T14:16:21.702 INFO:teuthology.orchestra.run.smithi049.stdout: CC oslib/strlcat.o 2024-01-18T14:16:21.747 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/mtd.o 2024-01-18T14:16:21.917 INFO:teuthology.orchestra.run.smithi049.stdout: CC oslib/libmtd.o 2024-01-18T14:16:22.306 INFO:teuthology.orchestra.run.smithi049.stdout: CC oslib/libmtd_legacy.o 2024-01-18T14:16:22.445 INFO:teuthology.orchestra.run.smithi049.stdout: CC diskutil.o 2024-01-18T14:16:22.661 INFO:teuthology.orchestra.run.smithi049.stdout: CC fifo.o 2024-01-18T14:16:22.717 INFO:teuthology.orchestra.run.smithi049.stdout: CC blktrace.o 2024-01-18T14:16:23.063 INFO:teuthology.orchestra.run.smithi049.stdout: CC cgroup.o 2024-01-18T14:16:23.215 INFO:teuthology.orchestra.run.smithi049.stdout: CC trim.o 2024-01-18T14:16:23.336 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/sg.o 2024-01-18T14:16:23.734 INFO:teuthology.orchestra.run.smithi049.stdout: CC oslib/linux-dev-lookup.o 2024-01-18T14:16:23.792 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/io_uring.o 2024-01-18T14:16:24.185 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/nvme.o 2024-01-18T14:16:24.498 INFO:teuthology.orchestra.run.smithi049.stdout: CC oslib/linux-blkzoned.o 2024-01-18T14:16:24.655 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/cmdprio.o 2024-01-18T14:16:24.902 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/rados.o 2024-01-18T14:16:25.084 INFO:teuthology.orchestra.run.smithi049.stdout: CC engines/rbd.o 2024-01-18T14:16:25.274 INFO:teuthology.orchestra.run.smithi049.stdout: CC fio.o 2024-01-18T14:16:25.365 INFO:teuthology.orchestra.run.smithi049.stdout: LINK fio 2024-01-18T14:16:25.559 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/log.o 2024-01-18T14:16:25.601 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/genzipf.o 2024-01-18T14:16:25.747 INFO:teuthology.orchestra.run.smithi049.stdout: CC oslib/strcasestr.o 2024-01-18T14:16:25.772 INFO:teuthology.orchestra.run.smithi049.stdout: CC oslib/strndup.o 2024-01-18T14:16:25.795 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/fio-genzipf 2024-01-18T14:16:25.892 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/btrace2fio.o 2024-01-18T14:16:26.320 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/fio-btrace2fio 2024-01-18T14:16:26.411 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/verify-state.o 2024-01-18T14:16:26.512 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/debug.o 2024-01-18T14:16:26.547 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/fio-verify-state 2024-01-18T14:16:26.638 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/stest.o 2024-01-18T14:16:26.709 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/arch.o 2024-01-18T14:16:26.739 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/stest 2024-01-18T14:16:26.833 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/ieee754.o 2024-01-18T14:16:26.875 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/ieee754 2024-01-18T14:16:26.964 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/axmap.o 2024-01-18T14:16:27.066 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/axmap 2024-01-18T14:16:27.157 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/lfsr-test.o 2024-01-18T14:16:27.255 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/lfsr-test 2024-01-18T14:16:27.350 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/gen-rand.o 2024-01-18T14:16:27.452 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/gen-rand 2024-01-18T14:16:27.547 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/memlock.o 2024-01-18T14:16:27.608 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/memlock 2024-01-18T14:16:27.699 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/read-to-pipe-async.o 2024-01-18T14:16:27.891 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/read-to-pipe-async 2024-01-18T14:16:27.981 INFO:teuthology.orchestra.run.smithi049.stdout: CC t/io_uring.o 2024-01-18T14:16:28.535 INFO:teuthology.orchestra.run.smithi049.stdout: LINK t/io_uring 2024-01-18T14:16:28.627 DEBUG:teuthology.orchestra.run.smithi049:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-01-18T14:16:28.659 DEBUG:teuthology.orchestra.run.smithi049:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-01-18T14:16:28.902 INFO:teuthology.orchestra.run.smithi049.stderr:14:16:28 - DEBUG - cbt - Settings.cluster: 2024-01-18T14:16:28.902 INFO:teuthology.orchestra.run.smithi049.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-01-18T14:16:28.903 INFO:teuthology.orchestra.run.smithi049.stderr: 'clients': ['ubuntu@smithi049.front.sepia.ceph.com'], 2024-01-18T14:16:28.903 INFO:teuthology.orchestra.run.smithi049.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-01-18T14:16:28.903 INFO:teuthology.orchestra.run.smithi049.stderr: 'head': 'ubuntu@smithi049.front.sepia.ceph.com', 2024-01-18T14:16:28.903 INFO:teuthology.orchestra.run.smithi049.stderr: 'is_teuthology': True, 2024-01-18T14:16:28.903 INFO:teuthology.orchestra.run.smithi049.stderr: 'iterations': 1, 2024-01-18T14:16:28.903 INFO:teuthology.orchestra.run.smithi049.stderr: 'mons': ['ubuntu@smithi049.front.sepia.ceph.com'], 2024-01-18T14:16:28.903 INFO:teuthology.orchestra.run.smithi049.stderr: 'osds': ['ubuntu@smithi049.front.sepia.ceph.com'], 2024-01-18T14:16:28.903 INFO:teuthology.orchestra.run.smithi049.stderr: 'osds_per_node': 3, 2024-01-18T14:16:28.903 INFO:teuthology.orchestra.run.smithi049.stderr: 'pid_dir': '/var/run/ceph', 2024-01-18T14:16:28.903 INFO:teuthology.orchestra.run.smithi049.stderr: 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 2024-01-18T14:16:28.903 INFO:teuthology.orchestra.run.smithi049.stderr: 'rebuild_every_test': False, 2024-01-18T14:16:28.903 INFO:teuthology.orchestra.run.smithi049.stderr: 'rgws': {'ubuntu@smithi049.front.sepia.ceph.com': None}, 2024-01-18T14:16:28.903 INFO:teuthology.orchestra.run.smithi049.stderr: 'tmp_dir': '/tmp/cbt', 2024-01-18T14:16:28.903 INFO:teuthology.orchestra.run.smithi049.stderr: 'use_existing': True, 2024-01-18T14:16:28.903 INFO:teuthology.orchestra.run.smithi049.stderr: 'user': 'ubuntu'} 2024-01-18T14:16:28.904 INFO:teuthology.orchestra.run.smithi049.stderr:14:16:28 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:16:28.904 INFO:teuthology.orchestra.run.smithi049.stderr:14:16:28 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:16:28.904 INFO:teuthology.orchestra.run.smithi049.stderr:14:16:28 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:16:28.904 INFO:teuthology.orchestra.run.smithi049.stderr:14:16:28 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread; fi 2024-01-18T14:16:28.906 INFO:teuthology.orchestra.run.smithi049.stderr:14:16:28 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:16:28.907 INFO:teuthology.orchestra.run.smithi049.stderr:14:16:28 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:16:28.907 INFO:teuthology.orchestra.run.smithi049.stderr:14:16:28 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread 2024-01-18T14:16:28.909 INFO:teuthology.orchestra.run.smithi049.stderr:14:16:28 - INFO - cbt - Pausing for 60s for idle monitoring. 2024-01-18T14:16:28.909 INFO:teuthology.orchestra.run.smithi049.stderr:14:16:28 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:16:28.909 INFO:teuthology.orchestra.run.smithi049.stderr:14:16:28 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:16:28.910 INFO:teuthology.orchestra.run.smithi049.stderr:14:16:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/idle_monitoring/collectl 2024-01-18T14:16:28.912 INFO:teuthology.orchestra.run.smithi049.stderr:14:16:28 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:16:28.912 INFO:teuthology.orchestra.run.smithi049.stderr:14:16:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-01-18T14:16:28.914 INFO:teuthology.orchestra.run.smithi049.stderr:14:16:28 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:16:28.914 INFO:teuthology.orchestra.run.smithi049.stderr:14:16:28 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:16:28.914 INFO:teuthology.orchestra.run.smithi049.stderr:14:16:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/idle_monitoring/perf 2024-01-18T14:16:28.917 INFO:teuthology.orchestra.run.smithi049.stderr:14:16:28 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:16:28.917 INFO:teuthology.orchestra.run.smithi049.stderr:14:16:28 - DEBUG - cbt - PerfMonitoring: in local_node 2024-01-18T14:16:28.917 INFO:teuthology.orchestra.run.smithi049.stderr:14:16:28 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-01-18T14:17:28.924 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:28 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:17:28.924 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:28 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:17:28.924 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-01-18T14:17:28.976 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:28 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:17:28.976 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:28 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:17:28.976 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-01-18T14:17:29.029 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:17:29.030 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:17:29.030 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/* 2024-01-18T14:17:29.051 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:17:29.051 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c/$(basename ${i}).smithi049.front.sepia.ceph.com ; done 2024-01-18T14:17:29.055 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:17:29.055 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:17:29.055 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/pool_monitoring/collectl 2024-01-18T14:17:29.058 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:17:29.058 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-01-18T14:17:29.060 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:17:29.060 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:17:29.060 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/pool_monitoring/perf 2024-01-18T14:17:29.063 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:17:29.063 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - PerfMonitoring: in local_node 2024-01-18T14:17:29.063 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-01-18T14:17:29.063 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:17:29.063 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:17:29.063 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete cbt-librbdfio cbt-librbdfio --yes-i-really-really-mean-it 2024-01-18T14:17:29.370 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:17:29.370 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:17:29.370 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create cbt-librbdfio 128 128 2024-01-18T14:17:29.730 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:17:29.730 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:17:29.730 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:29 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable cbt-librbdfio rbd 2024-01-18T14:17:30.734 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:30 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:17:30.734 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:30 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:17:30.734 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:30 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio size 3 2024-01-18T14:17:31.737 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:31 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:17:31.738 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:31 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:17:31.738 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:31 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio min_size 2 2024-01-18T14:17:32.743 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:32 - INFO - cbt - Checking Health after pool creation. 2024-01-18T14:17:37.749 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:37 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:17:37.750 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:37 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:17:37.750 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:37 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-01-18T14:17:38.078 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:38 - INFO - cbt - Final Pool Health Check. 2024-01-18T14:17:43.084 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:43 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:17:43.084 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:43 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:17:43.084 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:43 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-01-18T14:17:43.405 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:43 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:17:43.406 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:43 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:17:43.406 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:43 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=hostname -f 2024-01-18T14:17:43.408 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:43 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:17:43.408 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:43 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:17:43.408 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:43 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi049.front.sepia.ceph.com-0 --size 4096 --pool cbt-librbdfio --order 22 2024-01-18T14:17:45.102 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:45 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:17:45.102 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:45 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:17:45.102 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:45 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi049.front.sepia.ceph.com-1 --size 4096 --pool cbt-librbdfio --order 22 2024-01-18T14:17:45.197 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:45 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:17:45.197 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:45 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:17:45.198 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-01-18T14:17:45.227 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:45 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:17:45.228 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:45 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:17:45.228 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-01-18T14:17:45.274 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:45 - INFO - cbt - Attempting to populating fio files... 2024-01-18T14:17:45.274 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:45 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:17:45.274 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:45 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:17:45.274 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2024-01-18T14:17:45.276 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:45 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:17:45.276 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:45 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:17:45.276 INFO:teuthology.orchestra.run.smithi049.stderr:14:17:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2024-01-18T14:20:14.978 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:14 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:20:14.978 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:14 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c. 2024-01-18T14:20:14.978 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:14 - INFO - cbt - Setting OSD Read Ahead to: 4096 2024-01-18T14:20:14.978 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:14 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:20:14.978 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:14 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:20:14.978 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 4096 > /sys/block/'{}'/queue/read_ahead_kb" 2024-01-18T14:20:14.979 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:14 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread 2024-01-18T14:20:14.979 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:14 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:20:14.979 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:14 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:20:14.979 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread 2024-01-18T14:20:14.999 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:14 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:20:14.999 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:14 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:20:14.999 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-01-18T14:20:15.351 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:15 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:20:15.351 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-01-18T14:20:16.137 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:16 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:20:16.137 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:16 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:20:16.137 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:16 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread 2024-01-18T14:20:16.272 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:16 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:20:16.272 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:16 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:20:16.272 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:16 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/ceph_settings.out 2024-01-18T14:20:42.202 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:42 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:20:42.202 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:42 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:20:42.202 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo "[`date`] ceph progress" >> /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/pgautoscaler.log;/usr/bin/ceph -c /etc/ceph/ceph.conf progress | tee -a /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/pgautoscaler.log 2024-01-18T14:20:42.562 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:42 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:20:42.562 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:42 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:20:42.562 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/collectl 2024-01-18T14:20:42.565 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:42 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:20:42.565 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-01-18T14:20:42.566 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:42 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:20:42.567 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:42 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:20:42.567 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/perf 2024-01-18T14:20:42.570 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:42 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:20:42.570 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:42 - DEBUG - cbt - PerfMonitoring: in local_node 2024-01-18T14:20:42.570 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:42 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-01-18T14:20:42.570 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:42 - INFO - cbt - Running rbd fio randread test. 2024-01-18T14:20:42.570 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:42 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-0 2024-01-18T14:20:42.570 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:42 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:20:42.570 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:42 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:20:42.571 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=randread --output-format=json,normal --runtime=60 --numjobs=1 --direct=1 --bs=4096B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.0 2024-01-18T14:20:42.571 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:42 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-1 2024-01-18T14:20:42.572 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:42 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:20:42.572 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:42 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:20:42.572 INFO:teuthology.orchestra.run.smithi049.stderr:14:20:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=randread --output-format=json,normal --runtime=60 --numjobs=1 --direct=1 --bs=4096B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.1 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.1 2024-01-18T14:21:43.405 INFO:teuthology.orchestra.run.smithi049.stderr:14:21:43 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:21:43.406 INFO:teuthology.orchestra.run.smithi049.stderr:14:21:43 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:21:43.406 INFO:teuthology.orchestra.run.smithi049.stderr:14:21:43 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-01-18T14:21:43.431 INFO:teuthology.orchestra.run.smithi049.stderr:14:21:43 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:21:43.432 INFO:teuthology.orchestra.run.smithi049.stderr:14:21:43 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:21:43.432 INFO:teuthology.orchestra.run.smithi049.stderr:14:21:43 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-01-18T14:21:43.464 INFO:teuthology.orchestra.run.smithi049.stderr:14:21:43 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:21:43.464 INFO:teuthology.orchestra.run.smithi049.stderr:14:21:43 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/perf/perf.data 2024-01-18T14:21:43.465 INFO:teuthology.orchestra.run.smithi049.stderr:14:21:43 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:21:43.465 INFO:teuthology.orchestra.run.smithi049.stderr:14:21:43 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/perf/perf_stat.* 2024-01-18T14:21:43.467 INFO:teuthology.orchestra.run.smithi049.stderr:14:21:43 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:21:43.467 INFO:teuthology.orchestra.run.smithi049.stderr:14:21:43 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:21:43.467 INFO:teuthology.orchestra.run.smithi049.stderr:14:21:43 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/historic_ops.out 2024-01-18T14:21:43.775 INFO:teuthology.orchestra.run.smithi049.stderr:14:21:43 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:21:43.789 INFO:teuthology.orchestra.run.smithi049.stderr:14:21:43 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:21:43.789 INFO:teuthology.orchestra.run.smithi049.stderr:14:21:43 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/* 2024-01-18T14:21:43.809 INFO:teuthology.orchestra.run.smithi049.stderr:14:21:43 - DEBUG - cbt - get_fqdn_local()=smithi049.front.sepia.ceph.com 2024-01-18T14:21:43.809 INFO:teuthology.orchestra.run.smithi049.stderr:14:21:43 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c/$(basename ${i}).smithi049.front.sepia.ceph.com ; done 2024-01-18T14:21:43.846 INFO:teuthology.orchestra.run.smithi049.stderr:14:21:43 - INFO - cbt - Convert results to json format. 2024-01-18T14:21:43.846 INFO:teuthology.orchestra.run.smithi049.stderr:14:21:43 - DEBUG - cbt - Nodes : ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:21:43.847 INFO:teuthology.orchestra.run.smithi049.stdout:cleaning remote dir /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread 2024-01-18T14:21:43.847 INFO:teuthology.orchestra.run.smithi049.stdout:smithi049.front.sepia.ceph.com: smithi049.front.sepia.ceph.com 2024-01-18T14:21:43.847 INFO:teuthology.orchestra.run.smithi049.stdout: 2024-01-18T14:21:43.847 INFO:teuthology.orchestra.run.smithi049.stdout:['smithi049.front.sepia.ceph.com'] 2024-01-18T14:21:43.867 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2024-01-18T14:21:43.877 DEBUG:teuthology.orchestra.run.smithi049:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2024-01-18T14:21:44.030 DEBUG:teuthology.orchestra.run.smithi049:> rm --one-file-system -rf -- /home/ubuntu/cephtest/fio 2024-01-18T14:21:44.091 INFO:tasks.cbt_performance:Collecting CBT performance data config 2024-01-18T14:21:44.091 INFO:tasks.cbt_performance:reading cbt preferences 2024-01-18T14:21:44.091 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest -name 'json_output.*' 2024-01-18T14:21:44.157 INFO:teuthology.orchestra.run.smithi049.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c/json_output.0.smithi049.front.sepia.ceph.com 2024-01-18T14:21:44.157 INFO:teuthology.orchestra.run.smithi049.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c/json_output.1.smithi049.front.sepia.ceph.com 2024-01-18T14:21:44.158 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-01-18T14:21:44.158 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c/json_output.0.smithi049.front.sepia.ceph.com of=/dev/stdout 2024-01-18T14:21:44.183 INFO:tasks.cbt_performance:reading total cpu cycles 2024-01-18T14:21:44.183 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c -name 'perf_stat.*' 2024-01-18T14:21:44.238 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-01-18T14:21:44.238 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-01-18T14:21:44.238 DEBUG:teuthology.orchestra.run.smithi049:> set -ex 2024-01-18T14:21:44.238 DEBUG:teuthology.orchestra.run.smithi049:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c/json_output.1.smithi049.front.sepia.ceph.com of=/dev/stdout 2024-01-18T14:21:44.293 INFO:tasks.cbt_performance:reading total cpu cycles 2024-01-18T14:21:44.294 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c -name 'perf_stat.*' 2024-01-18T14:21:44.348 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-01-18T14:21:44.349 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-01-18T14:21:44.361 INFO:tasks.cbt_performance:Data inserted successfully. 2024-01-18T14:21:44.371 INFO:tasks.cbt_performance:Data inserted successfully. 2024-01-18T14:21:44.371 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-18T14:21:44.381 DEBUG:teuthology.orchestra.run.smithi049:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-01-18T14:21:44.503 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:21:44.495+0000 7f612a51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:21:44.505 INFO:teuthology.orchestra.run.smithi049.stderr:2024-01-18T14:21:44.503+0000 7f612a51e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-18T14:21:44.710 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-18T14:21:44.711 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-18T14:21:44.711 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-18T14:21:44.711 INFO:teuthology.orchestra.run:waiting for 300 2024-01-18T14:21:47.896 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-01-18T14:21:50.813 INFO:tasks.ceph.osd.0:Stopped 2024-01-18T14:21:50.813 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-18T14:21:50.813 INFO:teuthology.orchestra.run:waiting for 300 2024-01-18T14:21:53.199 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-01-18T14:21:53.199 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-01-18T14:21:56.914 INFO:tasks.ceph.osd.1:Stopped 2024-01-18T14:21:56.915 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-18T14:21:56.915 INFO:teuthology.orchestra.run:waiting for 300 2024-01-18T14:21:58.401 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-01-18T14:21:58.401 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-01-18T14:22:03.016 INFO:tasks.ceph.osd.2:Stopped 2024-01-18T14:22:03.016 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-18T14:22:03.016 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-18T14:22:03.016 INFO:teuthology.orchestra.run:waiting for 300 2024-01-18T14:22:03.049 INFO:tasks.ceph.mgr.x:Stopped 2024-01-18T14:22:03.049 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-18T14:22:03.049 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-18T14:22:03.050 INFO:teuthology.orchestra.run:waiting for 300 2024-01-18T14:22:03.103 INFO:tasks.ceph.mon.a:Stopped 2024-01-18T14:22:03.103 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-18T14:22:03.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v 'overall HEALTH' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-01-18T14:22:03.205 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:22:03.206 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-01-18T14:22:03.403 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-01-18T14:22:03.498 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:22:03.498 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-01-18T14:22:03.577 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:22:03.577 DEBUG:teuthology.orchestra.run.smithi049:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-01-18T14:22:03.626 INFO:tasks.ceph:Archiving mon data... 2024-01-18T14:22:03.627 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2024-01-18_13:55:25-crimson-rados-wip-matanb-crimson-testing-18.1-crimson-only-distro-crimson-smithi/7520293/data/mon.a.tgz 2024-01-18T14:22:03.628 DEBUG:teuthology.orchestra.run.smithi049:> mktemp 2024-01-18T14:22:03.644 INFO:teuthology.orchestra.run.smithi049.stdout:/tmp/tmp.q5Jw3s1Tx5 2024-01-18T14:22:03.644 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.q5Jw3s1Tx5 2024-01-18T14:22:03.779 DEBUG:teuthology.orchestra.run.smithi049:> sudo chmod 0666 /tmp/tmp.q5Jw3s1Tx5 2024-01-18T14:22:03.892 DEBUG:teuthology.orchestra.remote:smithi049:/tmp/tmp.q5Jw3s1Tx5 is 203KB 2024-01-18T14:22:03.945 DEBUG:teuthology.orchestra.run.smithi049:> rm -fr /tmp/tmp.q5Jw3s1Tx5 2024-01-18T14:22:03.959 INFO:tasks.ceph:Cleaning ceph cluster... 2024-01-18T14:22:03.960 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-18T14:22:04.099 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-01-18T14:22:04.100 INFO:tasks.ceph:Archiving crash dumps... 2024-01-18T14:22:04.102 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2024-01-18_13:55:25-crimson-rados-wip-matanb-crimson-testing-18.1-crimson-only-distro-crimson-smithi/7520293/remote/smithi049/crash 2024-01-18T14:22:04.103 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-18T14:22:04.170 INFO:tasks.ceph:Compressing logs... 2024-01-18T14:22:04.170 DEBUG:teuthology.orchestra.run.smithi049:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-18T14:22:04.259 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30196.log 2024-01-18T14:22:04.259 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30616.log 2024-01-18T14:22:04.260 INFO:teuthology.orchestra.run.smithi049.stderr:gzip/var/log/ceph/ceph-client.admin.30196.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30446.log 2024-01-18T14:22:04.260 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.30616.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-01-18T14:22:04.260 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.30446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34278.log 2024-01-18T14:22:04.260 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30320.log 2024-01-18T14:22:04.261 INFO:teuthology.orchestra.run.smithi049.stderr:gzip/var/log/ceph/ceph-client.admin.34278.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29210.log 2024-01-18T14:22:04.262 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.30320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34486.log 2024-01-18T14:22:04.262 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30319.log 2024-01-18T14:22:04.262 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.29210.log: /var/log/ceph/ceph-client.admin.34486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30636.log 2024-01-18T14:22:04.262 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.30319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29320.log 2024-01-18T14:22:04.262 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.30636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34342.log 2024-01-18T14:22:04.262 INFO:teuthology.orchestra.run.smithi049.stderr:gzip/var/log/ceph/ceph-client.admin.29320.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34487.log 2024-01-18T14:22:04.262 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.34342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30700.log 2024-01-18T14:22:04.262 INFO:teuthology.orchestra.run.smithi049.stderr:gzip/var/log/ceph/ceph-client.admin.34487.log: -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27762.log 2024-01-18T14:22:04.262 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.30700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29995.log 2024-01-18T14:22:04.270 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph.tmp-client.admin.27762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29558.log 2024-01-18T14:22:04.270 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.29995.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-18T14:22:04.271 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5/var/log/ceph/ceph-client.admin.29558.log: --verbose -- /var/log/ceph/ceph-client.admin.34310.log 2024-01-18T14:22:04.271 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34368.log 2024-01-18T14:22:04.271 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.34310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30784.log 2024-01-18T14:22:04.271 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30532.log/var/log/ceph/ceph-client.admin.34368.log: 2024-01-18T14:22:04.271 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.30784.log: -- /var/log/ceph/ceph-osd.1.log 2024-01-18T14:22:04.271 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.30532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30238.log 2024-01-18T14:22:04.271 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30406.log 2024-01-18T14:22:04.271 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-osd.1.log: gzip/var/log/ceph/ceph-client.admin.30238.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29400.log 2024-01-18T14:22:04.271 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.30406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29122.log 2024-01-18T14:22:04.271 INFO:teuthology.orchestra.run.smithi049.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30196.log.gz 2024-01-18T14:22:04.271 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.29400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30318.log 2024-01-18T14:22:04.271 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30446.log.gz 2024-01-18T14:22:04.271 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.29122.log: gzip -5 --verbose -- 49.6% /var/log/ceph/ceph-client.admin.29360.log 2024-01-18T14:22:04.271 INFO:teuthology.orchestra.run.smithi049.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30238.log.gz 2024-01-18T14:22:04.271 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.30318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30445.log 2024-01-18T14:22:04.272 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30319.log.gz 2024-01-18T14:22:04.272 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30320.log.gz 2024-01-18T14:22:04.272 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.29360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34709.log 2024-01-18T14:22:04.272 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30406.log.gz 2024-01-18T14:22:04.272 INFO:teuthology.orchestra.run.smithi049.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.30532.log.gz 2024-01-18T14:22:04.272 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5/var/log/ceph/ceph-client.admin.30445.log: --verbose -- /var/log/ceph/ceph-client.admin.34661.log 2024-01-18T14:22:04.272 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30318.log.gz 2024-01-18T14:22:04.272 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30616.log.gz 2024-01-18T14:22:04.272 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.34709.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30636.log.gz 2024-01-18T14:22:04.272 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34455.log 2024-01-18T14:22:04.272 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34709.log.gz 2024-01-18T14:22:04.272 INFO:teuthology.orchestra.run.smithi049.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30700.log.gz 2024-01-18T14:22:04.272 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.34661.log: gzip -5 --verbose 49.6% -- /var/log/ceph/ceph-mgr.x.log -- replaced with /var/log/ceph/ceph-client.admin.30445.log.gz 2024-01-18T14:22:04.272 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-01-18T14:22:04.272 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34661.log.gz 2024-01-18T14:22:04.273 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30784.log.gz 2024-01-18T14:22:04.273 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.34455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34414.log 2024-01-18T14:22:04.273 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34215.log 2024-01-18T14:22:04.273 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.34414.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-01-18T14:22:04.273 INFO:teuthology.orchestra.run.smithi049.stderr:gzip/var/log/ceph/ceph-client.admin.34215.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30154.log 2024-01-18T14:22:04.273 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-18T14:22:04.273 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-01-18T14:22:04.273 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.30154.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34708.log 2024-01-18T14:22:04.273 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34995.log 2024-01-18T14:22:04.273 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5/var/log/ceph/ceph-client.admin.34708.log: --verbose -- /var/log/ceph/ceph-client.admin.34391.log 2024-01-18T14:22:04.273 INFO:teuthology.orchestra.run.smithi049.stderr: 87.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-18T14:22:04.273 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34708.log.gz 2024-01-18T14:22:04.273 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.34995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30051.log 2024-01-18T14:22:04.273 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34995.log.gz 2024-01-18T14:22:04.273 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.34391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34438.log 2024-01-18T14:22:04.274 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.30051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29588.log 2024-01-18T14:22:04.274 INFO:teuthology.orchestra.run.smithi049.stderr:gzip/var/log/ceph/ceph-client.admin.34438.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30093.log 2024-01-18T14:22:04.274 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30552.log/var/log/ceph/ceph-client.admin.29588.log: 2024-01-18T14:22:04.274 INFO:teuthology.orchestra.run.smithi049.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30742.log 2024-01-18T14:22:04.275 INFO:teuthology.orchestra.run.smithi049.stderr:/var/log/ceph/ceph-client.admin.30093.log: /var/log/ceph/ceph-client.admin.30552.log: /var/log/ceph/ceph-client.admin.30742.log: 86.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-18T14:22:04.275 INFO:teuthology.orchestra.run.smithi049.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29995.log.gz 2024-01-18T14:22:04.275 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30051.log.gz 2024-01-18T14:22:04.275 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30093.log.gz 2024-01-18T14:22:04.275 INFO:teuthology.orchestra.run.smithi049.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30154.log.gz 2024-01-18T14:22:04.275 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30552.log.gz 2024-01-18T14:22:04.275 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30742.log.gz 2024-01-18T14:22:04.275 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29320.log.gz 2024-01-18T14:22:04.275 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29360.log.gz 2024-01-18T14:22:04.276 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29400.log.gz 2024-01-18T14:22:04.278 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29558.log.gz 2024-01-18T14:22:04.280 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29588.log.gz 2024-01-18T14:22:04.290 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34215.log.gz 2024-01-18T14:22:04.290 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34278.log.gz 2024-01-18T14:22:04.290 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34310.log.gz 2024-01-18T14:22:04.290 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34342.log.gz 2024-01-18T14:22:04.290 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34368.log.gz 2024-01-18T14:22:04.342 INFO:teuthology.orchestra.run.smithi049.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27762.log.gz 2024-01-18T14:22:04.351 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34391.log.gz 2024-01-18T14:22:04.351 INFO:teuthology.orchestra.run.smithi049.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34414.log.gz 2024-01-18T14:22:04.351 INFO:teuthology.orchestra.run.smithi049.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.34438.log.gz 2024-01-18T14:22:04.351 INFO:teuthology.orchestra.run.smithi049.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.34455.log.gz 2024-01-18T14:22:04.351 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34487.log.gz 2024-01-18T14:22:04.351 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34486.log.gz 2024-01-18T14:22:04.359 INFO:teuthology.orchestra.run.smithi049.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29122.log.gz 2024-01-18T14:22:04.359 INFO:teuthology.orchestra.run.smithi049.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29210.log.gz 2024-01-18T14:22:04.359 INFO:teuthology.orchestra.run.smithi049.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-01-18T14:22:04.427 INFO:teuthology.orchestra.run.smithi049.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-01-18T14:22:21.718 INFO:teuthology.orchestra.run.smithi049.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-01-18T14:22:21.765 INFO:teuthology.orchestra.run.smithi049.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-01-18T14:22:21.997 INFO:teuthology.orchestra.run.smithi049.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-01-18T14:22:21.999 INFO:teuthology.orchestra.run.smithi049.stderr: 2024-01-18T14:22:21.999 INFO:teuthology.orchestra.run.smithi049.stderr:real 0m17.775s 2024-01-18T14:22:21.999 INFO:teuthology.orchestra.run.smithi049.stderr:user 0m10.094s 2024-01-18T14:22:21.999 INFO:teuthology.orchestra.run.smithi049.stderr:sys 0m0.530s 2024-01-18T14:22:21.999 INFO:tasks.ceph:Archiving logs... 2024-01-18T14:22:22.000 DEBUG:teuthology.misc:Transferring archived files from smithi049:/var/log/ceph to /home/teuthworker/archive/matan-2024-01-18_13:55:25-crimson-rados-wip-matanb-crimson-testing-18.1-crimson-only-distro-crimson-smithi/7520293/remote/smithi049/log 2024-01-18T14:22:22.001 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-18T14:22:23.615 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-18T14:22:23.626 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-18T14:22:23.626 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-18T14:22:23.690 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-18T14:22:23.700 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-18T14:22:23.700 DEBUG:teuthology.orchestra.run.smithi049:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-18T14:22:23.718 INFO:teuthology.orchestra.run.smithi049.stderr:bash: line 1: ntpq: command not found 2024-01-18T14:22:23.789 INFO:teuthology.orchestra.run.smithi049.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-18T14:22:23.789 INFO:teuthology.orchestra.run.smithi049.stdout:=============================================================================== 2024-01-18T14:22:23.789 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 43 -591us[ -591us] +/- 95ms 2024-01-18T14:22:23.789 INFO:teuthology.orchestra.run.smithi049.stdout:^* hv02.front.sepia.ceph.com 3 6 377 43 -718us[ -720us] +/- 75ms 2024-01-18T14:22:23.789 INFO:teuthology.orchestra.run.smithi049.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +1284us[+1284us] +/- 82ms 2024-01-18T14:22:23.789 INFO:teuthology.orchestra.run.smithi049.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-18T14:22:23.801 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-18T14:22:23.810 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-18T14:22:23.811 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-18T14:22:23.820 DEBUG:teuthology.orchestra.run.smithi049:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-18T14:22:23.973 DEBUG:teuthology.orchestra.run.smithi049:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-18T14:22:24.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-18T14:22:24.006 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-18T14:22:24.016 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-18T14:22:24.026 INFO:teuthology.task.internal:Duration was 877.689895 seconds 2024-01-18T14:22:24.026 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-18T14:22:24.035 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-18T14:22:24.035 DEBUG:teuthology.orchestra.run.smithi049:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-18T14:22:24.192 INFO:teuthology.orchestra.run.smithi049.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-18T14:22:24.894 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-18T14:22:24.894 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi049.front.sepia.ceph.com 2024-01-18T14:22:24.894 DEBUG:teuthology.orchestra.run.smithi049:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-18T14:22:24.919 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-18T14:22:24.919 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-18T14:22:24.995 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-18T14:22:24.996 DEBUG:teuthology.orchestra.run.smithi049:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-18T14:22:25.160 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-18T14:22:25.170 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-18T14:22:25.170 DEBUG:teuthology.orchestra.run.smithi049:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-18T14:22:25.219 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-18T14:22:25.229 DEBUG:teuthology.orchestra.run.smithi049:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-18T14:22:25.274 INFO:teuthology.orchestra.run.smithi049.stdout:kernel.core_pattern = core 2024-01-18T14:22:25.294 DEBUG:teuthology.orchestra.run.smithi049:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-18T14:22:25.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-18T14:22:25.308 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-18T14:22:25.318 INFO:teuthology.task.internal:Transferring archived files... 2024-01-18T14:22:25.319 DEBUG:teuthology.misc:Transferring archived files from smithi049:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2024-01-18_13:55:25-crimson-rados-wip-matanb-crimson-testing-18.1-crimson-only-distro-crimson-smithi/7520293/remote/smithi049 2024-01-18T14:22:25.319 DEBUG:teuthology.orchestra.run.smithi049:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-18T14:22:25.429 INFO:teuthology.task.internal:Removing archive directory... 2024-01-18T14:22:25.429 DEBUG:teuthology.orchestra.run.smithi049:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-18T14:22:25.460 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-18T14:22:25.473 INFO:teuthology.task.internal:Not uploading archives. 2024-01-18T14:22:25.473 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-18T14:22:25.484 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-18T14:22:25.484 DEBUG:teuthology.orchestra.run.smithi049:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-18T14:22:25.515 INFO:teuthology.orchestra.run.smithi049.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 18 14:22 /home/ubuntu/cephtest 2024-01-18T14:22:25.516 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-18T14:22:25.528 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-18T14:22:25.551 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/2Jx1MmfVk/fio?orgId=1&var-branch_name=wip-matanb-crimson-testing-18.1-crimson-only&var-job_id_selected=7520293 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} duration: 877.6898953914642 flavor: crimson owner: scheduled_matan@teuthology success: true 2024-01-18T14:22:25.551 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-18T14:22:25.630 INFO:teuthology.run:pass