2023-11-08T12:13:36.767 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-08T12:13:36.768 DEBUG:teuthology.run:Teuthology command: teuthology --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} --verbose --archive /home/teuthworker/archive/matan-2023-11-08_11:46:17-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7451942 --name matan-2023-11-08_11:46:17-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi --owner scheduled_matan@teuthology -- /home/teuthworker/archive/matan-2023-11-08_11:46:17-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7451942/orig.config.yaml 2023-11-08T12:13:36.843 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T12:13:36.930 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2023-11-08_11:46:17-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7451942 branch: wip-matanb-crimson-do_osd_ops_execute-v3 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} email: null first_in_suite: false job_id: '7451942' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2023-11-08_11:46:17-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-do_osd_ops_execute-v3 ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: 25e18238395a1cded51f624e1e84ec38cbcb1b4a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 25e18238395a1cded51f624e1e84ec38cbcb1b4a workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: db6fbc96537be451d7eb9f6302778fac84588cb6 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 6890 sha1: 25e18238395a1cded51f624e1e84ec38cbcb1b4a sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_db6fbc96537be451d7eb9f6302778fac84588cb6/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: db6fbc96537be451d7eb9f6302778fac84588cb6 targets: smithi039.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE4V3tEy6fAGYFySNX4MVgLuAvkFpJGmhe6LcFBY8bcYpEM9N8OiG+RWqCOAapFLw4rsALyn0JrQSrzlsTWQ1x8= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4096 pool_monitoring_list: - collectl pool_profile: replicated run_monitoring_list: - collectl time: 60 write_only: true cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-08_11:46:17 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3590714 2023-11-08T12:13:36.930 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_db6fbc96537be451d7eb9f6302778fac84588cb6/qa; will attempt to use it 2023-11-08T12:13:36.931 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_db6fbc96537be451d7eb9f6302778fac84588cb6/qa/tasks 2023-11-08T12:13:36.932 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-08T12:13:36.934 INFO:teuthology.task.internal:Checking packages... 2023-11-08T12:13:36.963 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '25e18238395a1cded51f624e1e84ec38cbcb1b4a' 2023-11-08T12:13:36.964 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-08T12:13:36.964 INFO:teuthology.packaging:ref: None 2023-11-08T12:13:36.965 INFO:teuthology.packaging:tag: None 2023-11-08T12:13:36.965 INFO:teuthology.packaging:branch: wip-matanb-crimson-do_osd_ops_execute-v3 2023-11-08T12:13:36.966 INFO:teuthology.packaging:sha1: 25e18238395a1cded51f624e1e84ec38cbcb1b4a 2023-11-08T12:13:36.967 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-do_osd_ops_execute-v3 2023-11-08T12:13:37.180 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7227.g25e18238 2023-11-08T12:13:37.180 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-08T12:13:37.190 INFO:teuthology.task.internal:no buildpackages task found 2023-11-08T12:13:37.190 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-08T12:13:37.210 INFO:teuthology.task.internal:Saving configuration 2023-11-08T12:13:37.229 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-08T12:13:37.236 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-08T12:13:37.260 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-11-08_11:46:17-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7451942', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-08 12:07:48.806806', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE4V3tEy6fAGYFySNX4MVgLuAvkFpJGmhe6LcFBY8bcYpEM9N8OiG+RWqCOAapFLw4rsALyn0JrQSrzlsTWQ1x8='} 2023-11-08T12:13:37.261 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-08T12:13:37.267 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-11-08T12:13:37.267 INFO:teuthology.run_tasks:Running task console_log... 2023-11-08T12:13:37.320 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f81d7292940>, signals=[15]) 2023-11-08T12:13:37.321 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-08T12:13:37.327 INFO:teuthology.task.internal:Opening connections... 2023-11-08T12:13:37.327 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:13:37.329 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T12:13:37.429 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-08T12:13:37.436 DEBUG:teuthology.orchestra.run.smithi039:> uname -m 2023-11-08T12:13:37.503 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64 2023-11-08T12:13:37.503 DEBUG:teuthology.orchestra.run.smithi039:> cat /etc/os-release 2023-11-08T12:13:37.559 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="CentOS Stream" 2023-11-08T12:13:37.560 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="9" 2023-11-08T12:13:37.560 INFO:teuthology.orchestra.run.smithi039.stdout:ID="centos" 2023-11-08T12:13:37.560 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE="rhel fedora" 2023-11-08T12:13:37.560 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="9" 2023-11-08T12:13:37.561 INFO:teuthology.orchestra.run.smithi039.stdout:PLATFORM_ID="platform:el9" 2023-11-08T12:13:37.561 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-08T12:13:37.561 INFO:teuthology.orchestra.run.smithi039.stdout:ANSI_COLOR="0;31" 2023-11-08T12:13:37.561 INFO:teuthology.orchestra.run.smithi039.stdout:LOGO="fedora-logo-icon" 2023-11-08T12:13:37.562 INFO:teuthology.orchestra.run.smithi039.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-08T12:13:37.562 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://centos.org/" 2023-11-08T12:13:37.562 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-08T12:13:37.562 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-08T12:13:37.563 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-08T12:13:37.563 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2023-11-08T12:13:37.584 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-08T12:13:37.594 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-08T12:13:37.611 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-08T12:13:37.612 DEBUG:teuthology.orchestra.run.smithi039:> test '!' -e /home/ubuntu/cephtest 2023-11-08T12:13:37.629 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-08T12:13:37.636 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-08T12:13:37.636 DEBUG:teuthology.orchestra.run.smithi039:> test -z $(ls -A /var/lib/ceph) 2023-11-08T12:13:37.692 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-08T12:13:37.727 INFO:teuthology.run_tasks:Running task kernel... 2023-11-08T12:13:37.742 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-08T12:13:37.743 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-08T12:13:37.743 DEBUG:teuthology.orchestra.run.smithi039:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-08T12:13:37.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T12:13:37.761 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2023-11-08T12:13:37.818 INFO:teuthology.orchestra.run.smithi039.stdout:5.14.0-361.el9.x86_64 2023-11-08T12:13:37.819 INFO:teuthology.task.kernel:Running kernel on smithi039: 5.14.0-361.el9.x86_64 2023-11-08T12:13:37.819 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum install -y kernel 2023-11-08T12:13:39.812 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-9 - AppStream 16 MB/s | 6.3 MB 00:00 2023-11-08T12:13:41.187 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2023-11-08T12:13:42.128 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 7.9 MB 00:00 2023-11-08T12:13:57.027 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 18 MB 00:13 2023-11-08T12:14:01.186 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - CRB 8.1 MB/s | 5.8 MB 00:00 2023-11-08T12:14:03.670 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 15 kB 00:01 2023-11-08T12:14:07.197 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 5.7 MB/s | 19 MB 00:03 2023-11-08T12:14:12.055 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2023-11-08T12:14:14.536 INFO:teuthology.orchestra.run.smithi039.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-11-08T12:14:14.537 INFO:teuthology.orchestra.run.smithi039.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-11-08T12:14:14.637 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:14:14.649 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:14:14.649 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-11-08T12:14:14.650 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:14:14.650 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-11-08T12:14:14.650 INFO:teuthology.orchestra.run.smithi039.stdout: kernel x86_64 5.14.0-382.el9 CentOS-BaseOS 5.4 M 2023-11-08T12:14:14.650 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-11-08T12:14:14.651 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-core x86_64 5.14.0-382.el9 CentOS-BaseOS 20 M 2023-11-08T12:14:14.651 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-modules x86_64 5.14.0-382.el9 CentOS-BaseOS 38 M 2023-11-08T12:14:14.651 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-modules-core x86_64 5.14.0-382.el9 CentOS-BaseOS 32 M 2023-11-08T12:14:14.651 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:14:14.652 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-11-08T12:14:14.652 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:14:14.652 INFO:teuthology.orchestra.run.smithi039.stdout:Install 4 Packages 2023-11-08T12:14:14.652 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:14:14.653 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 95 M 2023-11-08T12:14:14.654 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 121 M 2023-11-08T12:14:14.654 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-11-08T12:14:14.881 INFO:teuthology.orchestra.run.smithi039.stdout:(1/4): kernel-5.14.0-382.el9.x86_64.rpm 24 MB/s | 5.4 MB 00:00 2023-11-08T12:14:15.157 INFO:teuthology.orchestra.run.smithi039.stdout:(2/4): kernel-core-5.14.0-382.el9.x86_64.rpm 40 MB/s | 20 MB 00:00 2023-11-08T12:14:15.515 INFO:teuthology.orchestra.run.smithi039.stdout:(3/4): kernel-modules-core-5.14.0-382.el9.x86_6 51 MB/s | 32 MB 00:00 2023-11-08T12:14:15.907 INFO:teuthology.orchestra.run.smithi039.stdout:(4/4): kernel-modules-5.14.0-382.el9.x86_64.rpm 30 MB/s | 38 MB 00:01 2023-11-08T12:14:15.911 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:14:15.911 INFO:teuthology.orchestra.run.smithi039.stdout:Total 76 MB/s | 95 MB 00:01 2023-11-08T12:14:15.911 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-11-08T12:14:16.362 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-11-08T12:14:16.363 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-11-08T12:14:17.392 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-11-08T12:14:17.392 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-11-08T12:14:19.874 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-11-08T12:14:20.569 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : kernel-modules-core-5.14.0-382.el9.x86_64 1/4 2023-11-08T12:14:20.690 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : kernel-core-5.14.0-382.el9.x86_64 2/4 2023-11-08T12:14:23.553 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-core-5.14.0-382.el9.x86_64 2/4 2023-11-08T12:14:23.734 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : kernel-modules-5.14.0-382.el9.x86_64 3/4 2023-11-08T12:14:28.678 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-modules-5.14.0-382.el9.x86_64 3/4 2023-11-08T12:14:28.778 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : kernel-5.14.0-382.el9.x86_64 4/4 2023-11-08T12:14:33.737 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-modules-core-5.14.0-382.el9.x86_64 4/4 2023-11-08T12:15:04.237 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-core-5.14.0-382.el9.x86_64 4/4 2023-11-08T12:15:04.282 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-modules-5.14.0-382.el9.x86_64 4/4 2023-11-08T12:15:04.751 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: kernel-5.14.0-382.el9.x86_64 4/4 2023-11-08T12:15:04.752 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : kernel-5.14.0-382.el9.x86_64 1/4 2023-11-08T12:15:04.752 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : kernel-core-5.14.0-382.el9.x86_64 2/4 2023-11-08T12:15:04.752 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : kernel-modules-5.14.0-382.el9.x86_64 3/4 2023-11-08T12:15:05.157 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : kernel-modules-core-5.14.0-382.el9.x86_64 4/4 2023-11-08T12:15:05.157 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:15:05.158 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-11-08T12:15:05.158 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-5.14.0-382.el9.x86_64 2023-11-08T12:15:05.158 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-core-5.14.0-382.el9.x86_64 2023-11-08T12:15:05.158 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-modules-5.14.0-382.el9.x86_64 2023-11-08T12:15:05.159 INFO:teuthology.orchestra.run.smithi039.stdout: kernel-modules-core-5.14.0-382.el9.x86_64 2023-11-08T12:15:05.159 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:15:05.159 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:15:05.662 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q kernel | sort -rV | head -n 1 2023-11-08T12:15:05.831 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-5.14.0-382.el9.x86_64 2023-11-08T12:15:05.831 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-382.el9.x86_64 2023-11-08T12:15:05.831 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-382.el9.x86_64 2023-11-08T12:15:05.832 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-08T12:15:05.832 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-08T12:15:05.832 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-11-08T12:15:05.832 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi039.front.sepia.ceph.com, path=None, version=distro) 2023-11-08T12:15:05.833 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q kernel | sort -rV | head -n 1 2023-11-08T12:15:05.943 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-5.14.0-382.el9.x86_64 2023-11-08T12:15:05.943 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-382.el9.x86_64 2023-11-08T12:15:05.943 DEBUG:teuthology.orchestra.run.smithi039:> sudo rpm -qi grub2-tools 2023-11-08T12:15:05.981 INFO:teuthology.orchestra.run.smithi039.stdout:Name : grub2-tools 2023-11-08T12:15:05.981 INFO:teuthology.orchestra.run.smithi039.stdout:Epoch : 1 2023-11-08T12:15:05.981 INFO:teuthology.orchestra.run.smithi039.stdout:Version : 2.06 2023-11-08T12:15:05.981 INFO:teuthology.orchestra.run.smithi039.stdout:Release : 68.el9 2023-11-08T12:15:05.982 INFO:teuthology.orchestra.run.smithi039.stdout:Architecture: x86_64 2023-11-08T12:15:05.982 INFO:teuthology.orchestra.run.smithi039.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-11-08T12:15:05.982 INFO:teuthology.orchestra.run.smithi039.stdout:Group : Unspecified 2023-11-08T12:15:05.982 INFO:teuthology.orchestra.run.smithi039.stdout:Size : 8264254 2023-11-08T12:15:05.983 INFO:teuthology.orchestra.run.smithi039.stdout:License : GPLv3+ 2023-11-08T12:15:05.983 INFO:teuthology.orchestra.run.smithi039.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-11-08T12:15:05.983 INFO:teuthology.orchestra.run.smithi039.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-11-08T12:15:05.983 INFO:teuthology.orchestra.run.smithi039.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-11-08T12:15:05.984 INFO:teuthology.orchestra.run.smithi039.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-11-08T12:15:05.984 INFO:teuthology.orchestra.run.smithi039.stdout:Packager : builder@centos.org 2023-11-08T12:15:05.984 INFO:teuthology.orchestra.run.smithi039.stdout:Vendor : CentOS 2023-11-08T12:15:05.984 INFO:teuthology.orchestra.run.smithi039.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-08T12:15:05.984 INFO:teuthology.orchestra.run.smithi039.stdout:Summary : Support tools for GRUB. 2023-11-08T12:15:05.985 INFO:teuthology.orchestra.run.smithi039.stdout:Description : 2023-11-08T12:15:05.985 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:15:05.985 INFO:teuthology.orchestra.run.smithi039.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-08T12:15:05.985 INFO:teuthology.orchestra.run.smithi039.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-08T12:15:05.985 INFO:teuthology.orchestra.run.smithi039.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-08T12:15:05.986 INFO:teuthology.orchestra.run.smithi039.stdout:hardware devices. 2023-11-08T12:15:05.986 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:15:05.986 INFO:teuthology.orchestra.run.smithi039.stdout:This subpackage provides tools for support of all platforms. 2023-11-08T12:15:05.987 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-08T12:15:05.988 INFO:teuthology.task.kernel:Updating grub on smithi039 to boot 5.14.0-382.el9.x86_64 2023-11-08T12:15:05.988 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-08T12:15:06.574 INFO:teuthology.orchestra.run.smithi039.stderr:Generating grub configuration file ... 2023-11-08T12:15:07.970 INFO:teuthology.orchestra.run.smithi039.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-11-08T12:15:08.016 INFO:teuthology.orchestra.run.smithi039.stderr:done 2023-11-08T12:15:08.018 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2023-11-08T12:15:08.034 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.jn5svHSnsT 2023-11-08T12:15:08.035 DEBUG:teuthology.orchestra.run.smithi039:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.jn5svHSnsT 2023-11-08T12:15:08.107 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.jn5svHSnsT 2023-11-08T12:15:08.260 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.jn5svHSnsT is 6KB 2023-11-08T12:15:08.309 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.jn5svHSnsT 2023-11-08T12:15:08.324 DEBUG:teuthology.orchestra.run.smithi039:> sudo /bin/ls /boot/loader/entries || true 2023-11-08T12:15:08.392 INFO:teuthology.orchestra.run.smithi039.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-11-08T12:15:08.392 INFO:teuthology.orchestra.run.smithi039.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-11-08T12:15:08.393 INFO:teuthology.orchestra.run.smithi039.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-11-08T12:15:08.393 INFO:teuthology.orchestra.run.smithi039.stdout:15949e3c88704300882482eddfd7baae-5.14.0-382.el9.x86_64.conf 2023-11-08T12:15:08.394 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-382.el9.x86_64 2023-11-08T12:15:08.547 DEBUG:teuthology.orchestra.run.smithi039:> sudo shutdown -r now 2023-11-08T12:15:38.579 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-11-08T12:15:38.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-11-08T12:15:38.581 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T12:15:57.051 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2023-11-08T12:16:06.059 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-11-08T12:16:06.060 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T12:16:09.114 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2023-11-08T12:16:21.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-11-08T12:16:21.128 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T12:16:24.186 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2023-11-08T12:16:39.202 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-11-08T12:16:39.203 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T12:17:39.264 DEBUG:teuthology.orchestra.remote:timed out 2023-11-08T12:17:57.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-11-08T12:17:57.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T12:17:57.689 DEBUG:teuthology.orchestra.run.smithi039:> true 2023-11-08T12:17:58.056 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-11-08T12:17:58.057 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-382.el9.x86_64"... 2023-11-08T12:17:58.057 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2023-11-08T12:17:58.072 INFO:teuthology.orchestra.run.smithi039.stdout:5.14.0-382.el9.x86_64 2023-11-08T12:17:58.073 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-382.el9.x86_64 vs 5.14.0-382.el9.x86_64 2023-11-08T12:17:58.073 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-08T12:17:58.073 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-08T12:17:59.075 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-11-08T12:17:59.075 DEBUG:teuthology.orchestra.run.smithi039:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-08T12:17:59.170 INFO:teuthology.orchestra.run.smithi039.stdout:ttyS1 2023-11-08T12:17:59.199 DEBUG:teuthology.parallel:result is None 2023-11-08T12:17:59.200 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-08T12:17:59.207 INFO:teuthology.task.internal:Creating test directory... 2023-11-08T12:17:59.208 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-08T12:17:59.225 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-08T12:17:59.257 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-08T12:17:59.290 INFO:teuthology.task.internal:Creating archive directory... 2023-11-08T12:17:59.291 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-08T12:17:59.320 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-08T12:17:59.328 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-08T12:17:59.328 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-08T12:17:59.400 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-08T12:17:59.415 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-08T12:17:59.418 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-08T12:17:59.427 INFO:teuthology.task.internal:Configuring sudo... 2023-11-08T12:17:59.428 DEBUG:teuthology.orchestra.run.smithi039:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-08T12:17:59.492 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-08T12:17:59.501 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-08T12:17:59.502 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-08T12:17:59.549 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-08T12:17:59.636 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-08T12:17:59.690 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-08T12:17:59.775 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-08T12:17:59.821 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-11-08T12:17:59.821 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-08T12:17:59.892 DEBUG:teuthology.orchestra.run.smithi039:> sudo service rsyslog restart 2023-11-08T12:17:59.979 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-08T12:18:00.470 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-08T12:18:00.479 INFO:teuthology.task.internal:Starting timer... 2023-11-08T12:18:00.479 INFO:teuthology.run_tasks:Running task pcp... 2023-11-08T12:18:00.535 INFO:teuthology.run_tasks:Running task selinux... 2023-11-08T12:18:00.568 DEBUG:teuthology.orchestra.run.smithi039:> sudo service auditd rotate 2023-11-08T12:18:00.627 INFO:teuthology.orchestra.run.smithi039.stdout:Rotating logs: 2023-11-08T12:18:00.629 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-08T12:18:00.629 DEBUG:teuthology.orchestra.run.smithi039:> /usr/sbin/getenforce 2023-11-08T12:18:00.695 INFO:teuthology.orchestra.run.smithi039.stdout:Permissive 2023-11-08T12:18:00.695 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi039.front.sepia.ceph.com': 'permissive'} 2023-11-08T12:18:00.696 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-08T12:18:00.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T12:18:00.728 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-08T12:18:00.728 DEBUG:teuthology.orchestra.run.smithi039:> sudo /usr/sbin/setenforce permissive 2023-11-08T12:18:00.804 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-08T12:18:00.814 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-08T12:18:00.815 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-08T12:18:00.834 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-08T12:18:00.835 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi039.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-08T12:23:43.426 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi039.front.sepia.ceph.com')] 2023-11-08T12:23:43.428 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-11-08T12:23:43.429 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-08T12:23:43.526 DEBUG:teuthology.orchestra.run.smithi039:> true 2023-11-08T12:23:43.592 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-11-08T12:23:43.593 INFO:teuthology.run_tasks:Running task clock... 2023-11-08T12:23:43.605 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-08T12:23:43.606 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-08T12:23:43.606 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-08T12:23:43.667 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-08T12:23:43.687 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-08T12:23:43.724 INFO:teuthology.orchestra.run.smithi039.stderr:sudo: ntpd: command not found 2023-11-08T12:23:43.742 INFO:teuthology.orchestra.run.smithi039.stdout:506 Cannot talk to daemon 2023-11-08T12:23:43.762 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-08T12:23:43.782 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-08T12:23:43.829 INFO:teuthology.orchestra.run.smithi039.stderr:bash: line 1: ntpq: command not found 2023-11-08T12:23:43.833 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-08T12:23:43.833 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2023-11-08T12:23:43.833 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T12:23:43.833 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T12:23:43.834 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T12:23:43.834 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T12:23:43.834 INFO:teuthology.run_tasks:Running task install... 2023-11-08T12:23:43.846 DEBUG:teuthology.task.install:project ceph 2023-11-08T12:23:43.847 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '25e18238395a1cded51f624e1e84ec38cbcb1b4a'}} 2023-11-08T12:23:43.848 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '25e18238395a1cded51f624e1e84ec38cbcb1b4a'} 2023-11-08T12:23:43.848 INFO:teuthology.task.install:Using flavor: crimson 2023-11-08T12:23:43.854 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-08T12:23:43.854 INFO:teuthology.task.install:extra packages: [] 2023-11-08T12:23:43.855 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '25e18238395a1cded51f624e1e84ec38cbcb1b4a', 'tag': None, 'wait_for_package': False} 2023-11-08T12:23:43.855 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=25e18238395a1cded51f624e1e84ec38cbcb1b4a 2023-11-08T12:23:44.003 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-do_osd_ops_execute-v3/25e18238395a1cded51f624e1e84ec38cbcb1b4a/centos/9/flavors/crimson/ 2023-11-08T12:23:44.003 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7227.g25e18238 2023-11-08T12:23:44.180 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-do_osd_ops_execute-v3/25e18238395a1cded51f624e1e84ec38cbcb1b4a/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-do_osd_ops_execute-v3/25e18238395a1cded51f624e1e84ec38cbcb1b4a/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-matanb-crimson-do_osd_ops_execute-v3/25e18238395a1cded51f624e1e84ec38cbcb1b4a/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-08T12:23:44.181 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-11-08T12:23:44.181 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-08T12:23:44.213 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-08T12:23:44.213 DEBUG:teuthology.orchestra.run.smithi039:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/25e18238395a1cded51f624e1e84ec38cbcb1b4a/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-08T12:23:44.285 DEBUG:teuthology.orchestra.run.smithi039:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-08T12:23:44.372 DEBUG:teuthology.orchestra.run.smithi039:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-08T12:23:44.402 INFO:teuthology.orchestra.run.smithi039.stdout:check_obsoletes = 1 2023-11-08T12:23:44.404 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum clean all 2023-11-08T12:23:44.657 INFO:teuthology.orchestra.run.smithi039.stdout:72 files removed 2023-11-08T12:23:44.684 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-radosgw 2023-11-08T12:23:45.522 INFO:teuthology.orchestra.run.smithi039.stdout:ceph packages for x86_64 163 kB/s | 81 kB 00:00 2023-11-08T12:23:45.956 INFO:teuthology.orchestra.run.smithi039.stdout:ceph noarch packages 25 kB/s | 9.5 kB 00:00 2023-11-08T12:23:46.373 INFO:teuthology.orchestra.run.smithi039.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2023-11-08T12:23:46.767 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-9 - AppStream 18 MB/s | 6.3 MB 00:00 2023-11-08T12:23:48.110 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-9 - BaseOS 7.8 MB/s | 2.0 MB 00:00 2023-11-08T12:23:52.467 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 7.9 MB 00:04 2023-11-08T12:23:56.146 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - AppStream 7.1 MB/s | 18 MB 00:02 2023-11-08T12:24:05.644 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - CRB 998 kB/s | 5.8 MB 00:05 2023-11-08T12:24:08.011 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 15 kB 00:01 2023-11-08T12:24:10.754 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 7.5 MB/s | 19 MB 00:02 2023-11-08T12:24:15.554 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2023-11-08T12:24:18.316 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:24:18.327 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:24:18.327 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-11-08T12:24:18.327 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:24:18.328 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-11-08T12:24:18.328 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw x86_64 2:18.0.0-7227.g25e18238.el9 ceph 8.4 M 2023-11-08T12:24:18.328 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrading: 2023-11-08T12:24:18.328 INFO:teuthology.orchestra.run.smithi039.stdout: librados2 x86_64 2:18.0.0-7227.g25e18238.el9 ceph 3.1 M 2023-11-08T12:24:18.328 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1 x86_64 2:18.0.0-7227.g25e18238.el9 ceph 3.0 M 2023-11-08T12:24:18.329 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-11-08T12:24:18.329 INFO:teuthology.orchestra.run.smithi039.stdout: boost-program-options 2023-11-08T12:24:18.329 INFO:teuthology.orchestra.run.smithi039.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-11-08T12:24:18.329 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base x86_64 2:18.0.0-7227.g25e18238.el9 ceph 6.6 M 2023-11-08T12:24:18.329 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common x86_64 2:18.0.0-7227.g25e18238.el9 ceph 17 M 2023-11-08T12:24:18.330 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux x86_64 2:18.0.0-7227.g25e18238.el9 ceph 24 k 2023-11-08T12:24:18.330 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-11-08T12:24:18.330 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-11-08T12:24:18.330 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-11-08T12:24:18.330 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2 x86_64 2:18.0.0-7227.g25e18238.el9 ceph 728 k 2023-11-08T12:24:18.331 INFO:teuthology.orchestra.run.smithi039.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-11-08T12:24:18.331 INFO:teuthology.orchestra.run.smithi039.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-11-08T12:24:18.331 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-11-08T12:24:18.331 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1 x86_64 2:18.0.0-7227.g25e18238.el9 ceph 470 k 2023-11-08T12:24:18.331 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-11-08T12:24:18.332 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2 x86_64 2:18.0.0-7227.g25e18238.el9 ceph 4.5 M 2023-11-08T12:24:18.332 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-11-08T12:24:18.332 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-11-08T12:24:18.333 INFO:teuthology.orchestra.run.smithi039.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-11-08T12:24:18.333 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse 2023-11-08T12:24:18.333 INFO:teuthology.orchestra.run.smithi039.stdout: x86_64 2:18.0.0-7227.g25e18238.el9 ceph 45 k 2023-11-08T12:24:18.333 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common x86_64 2:18.0.0-7227.g25e18238.el9 ceph 127 k 2023-11-08T12:24:18.333 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs x86_64 2:18.0.0-7227.g25e18238.el9 ceph 162 k 2023-11-08T12:24:18.334 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-11-08T12:24:18.334 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados x86_64 2:18.0.0-7227.g25e18238.el9 ceph 320 k 2023-11-08T12:24:18.334 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd x86_64 2:18.0.0-7227.g25e18238.el9 ceph 298 k 2023-11-08T12:24:18.334 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw x86_64 2:18.0.0-7227.g25e18238.el9 ceph 99 k 2023-11-08T12:24:18.334 INFO:teuthology.orchestra.run.smithi039.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-11-08T12:24:18.335 INFO:teuthology.orchestra.run.smithi039.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-11-08T12:24:18.335 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:24:18.335 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-11-08T12:24:18.335 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:24:18.335 INFO:teuthology.orchestra.run.smithi039.stdout:Install 27 Packages 2023-11-08T12:24:18.336 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrade 2 Packages 2023-11-08T12:24:18.336 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:24:18.336 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 54 M 2023-11-08T12:24:18.337 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-11-08T12:24:19.356 INFO:teuthology.orchestra.run.smithi039.stdout:(1/29): ceph-radosgw-18.0.0-7227.g25e18238.el9. 10 MB/s | 8.4 MB 00:00 2023-11-08T12:24:19.414 INFO:teuthology.orchestra.run.smithi039.stdout:(2/29): ceph-selinux-18.0.0-7227.g25e18238.el9. 413 kB/s | 24 kB 00:00 2023-11-08T12:24:19.514 INFO:teuthology.orchestra.run.smithi039.stdout:(3/29): libcephfs2-18.0.0-7227.g25e18238.el9.x8 7.1 MB/s | 728 kB 00:00 2023-11-08T12:24:19.598 INFO:teuthology.orchestra.run.smithi039.stdout:(4/29): libradosstriper1-18.0.0-7227.g25e18238. 5.5 MB/s | 470 kB 00:00 2023-11-08T12:24:20.048 INFO:teuthology.orchestra.run.smithi039.stdout:(5/29): ceph-base-18.0.0-7227.g25e18238.el9.x86 4.4 MB/s | 6.6 MB 00:01 2023-11-08T12:24:20.123 INFO:teuthology.orchestra.run.smithi039.stdout:(6/29): librgw2-18.0.0-7227.g25e18238.el9.x86_6 8.6 MB/s | 4.5 MB 00:00 2023-11-08T12:24:20.237 INFO:teuthology.orchestra.run.smithi039.stdout:(7/29): python3-ceph-argparse-18.0.0-7227.g25e1 239 kB/s | 45 kB 00:00 2023-11-08T12:24:20.257 INFO:teuthology.orchestra.run.smithi039.stdout:(8/29): python3-ceph-common-18.0.0-7227.g25e182 948 kB/s | 127 kB 00:00 2023-11-08T12:24:20.308 INFO:teuthology.orchestra.run.smithi039.stdout:(9/29): python3-cephfs-18.0.0-7227.g25e18238.el 2.2 MB/s | 162 kB 00:00 2023-11-08T12:24:20.333 INFO:teuthology.orchestra.run.smithi039.stdout:(10/29): python3-rados-18.0.0-7227.g25e18238.el 4.1 MB/s | 320 kB 00:00 2023-11-08T12:24:20.408 INFO:teuthology.orchestra.run.smithi039.stdout:(11/29): python3-rbd-18.0.0-7227.g25e18238.el9. 2.9 MB/s | 298 kB 00:00 2023-11-08T12:24:20.434 INFO:teuthology.orchestra.run.smithi039.stdout:(12/29): python3-rgw-18.0.0-7227.g25e18238.el9. 991 kB/s | 99 kB 00:00 2023-11-08T12:24:20.609 INFO:teuthology.orchestra.run.smithi039.stdout:(13/29): ceph-common-18.0.0-7227.g25e18238.el9. 8.3 MB/s | 17 MB 00:02 2023-11-08T12:24:20.735 INFO:teuthology.orchestra.run.smithi039.stdout:(14/29): boost-program-options-1.75.0-8.el9.x86 325 kB/s | 106 kB 00:00 2023-11-08T12:24:20.759 INFO:teuthology.orchestra.run.smithi039.stdout:(15/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 489 kB/s | 159 kB 00:00 2023-11-08T12:24:20.785 INFO:teuthology.orchestra.run.smithi039.stdout:(16/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2023-11-08T12:24:20.802 INFO:teuthology.orchestra.run.smithi039.stdout:(17/29): python3-prettytable-0.7.2-27.el9.noarc 994 kB/s | 41 kB 00:00 2023-11-08T12:24:20.836 INFO:teuthology.orchestra.run.smithi039.stdout:(18/29): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 661 kB 00:00 2023-11-08T12:24:20.928 INFO:teuthology.orchestra.run.smithi039.stdout:(19/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 318 kB/s | 45 kB 00:00 2023-11-08T12:24:21.386 INFO:teuthology.orchestra.run.smithi039.stdout:(20/29): libarrow-doc-9.0.0-9.el9.noarch.rpm 53 kB/s | 24 kB 00:00 2023-11-08T12:24:21.612 INFO:teuthology.orchestra.run.smithi039.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 216 kB/s | 49 kB 00:00 2023-11-08T12:24:21.637 INFO:teuthology.orchestra.run.smithi039.stdout:(22/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 370 kB/s | 309 kB 00:00 2023-11-08T12:24:21.770 INFO:teuthology.orchestra.run.smithi039.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 425 kB/s | 67 kB 00:00 2023-11-08T12:24:21.987 INFO:teuthology.orchestra.run.smithi039.stdout:(24/29): libarrow-9.0.0-9.el9.x86_64.rpm 3.8 MB/s | 4.4 MB 00:01 2023-11-08T12:24:22.013 INFO:teuthology.orchestra.run.smithi039.stdout:(25/29): parquet-libs-9.0.0-9.el9.x86_64.rpm 2.2 MB/s | 837 kB 00:00 2023-11-08T12:24:22.038 INFO:teuthology.orchestra.run.smithi039.stdout:(26/29): re2-20211101-3.el9.x86_64.rpm 717 kB/s | 192 kB 00:00 2023-11-08T12:24:22.147 INFO:teuthology.orchestra.run.smithi039.stdout:(27/29): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2023-11-08T12:24:22.514 INFO:teuthology.orchestra.run.smithi039.stdout:(28/29): librados2-18.0.0-7227.g25e18238.el9.x8 6.2 MB/s | 3.1 MB 00:00 2023-11-08T12:24:22.564 INFO:teuthology.orchestra.run.smithi039.stdout:(29/29): librbd1-18.0.0-7227.g25e18238.el9.x86_ 5.8 MB/s | 3.0 MB 00:00 2023-11-08T12:24:22.571 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:24:22.571 INFO:teuthology.orchestra.run.smithi039.stdout:Total 13 MB/s | 54 MB 00:04 2023-11-08T12:24:22.584 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-11-08T12:24:22.627 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-11-08T12:24:22.627 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-11-08T12:24:23.117 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-11-08T12:24:23.117 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-11-08T12:24:24.521 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-11-08T12:24:24.949 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-11-08T12:24:24.974 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librados2-2:18.0.0-7227.g25e18238.el9.x86_64 2/31 2023-11-08T12:24:25.092 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-2:18.0.0-7227.g25e18238.el9.x86_64 2/31 2023-11-08T12:24:25.189 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rados-2:18.0.0-7227.g25e18238.el9.x86_64 3/31 2023-11-08T12:24:25.234 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs2-2:18.0.0-7227.g25e18238.el9.x86_64 4/31 2023-11-08T12:24:25.289 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephfs2-2:18.0.0-7227.g25e18238.el9.x86_64 4/31 2023-11-08T12:24:25.340 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/31 2023-11-08T12:24:25.372 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/31 2023-11-08T12:24:25.402 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/31 2023-11-08T12:24:25.505 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-argparse-2:18.0.0-7227.g25e18238.el9. 8/31 2023-11-08T12:24:25.573 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cephfs-2:18.0.0-7227.g25e18238.el9.x86_64 9/31 2023-11-08T12:24:25.596 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libradosstriper1-2:18.0.0-7227.g25e18238.el9.x86_6 10/31 2023-11-08T12:24:25.685 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7227.g25e18238.el9.x86_6 10/31 2023-11-08T12:24:25.729 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/31 2023-11-08T12:24:25.839 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-11-08T12:24:25.872 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/31 2023-11-08T12:24:25.912 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-11-08T12:24:25.944 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-11-08T12:24:25.974 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/31 2023-11-08T12:24:26.006 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-11-08T12:24:26.336 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-11-08T12:24:26.361 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librbd1-2:18.0.0-7227.g25e18238.el9.x86_64 19/31 2023-11-08T12:24:26.452 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-2:18.0.0-7227.g25e18238.el9.x86_64 19/31 2023-11-08T12:24:26.482 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rbd-2:18.0.0-7227.g25e18238.el9.x86_64 20/31 2023-11-08T12:24:26.519 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-11-08T12:24:26.973 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 22/31 2023-11-08T12:24:27.003 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librgw2-2:18.0.0-7227.g25e18238.el9.x86_64 23/31 2023-11-08T12:24:27.075 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librgw2-2:18.0.0-7227.g25e18238.el9.x86_64 23/31 2023-11-08T12:24:27.154 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rgw-2:18.0.0-7227.g25e18238.el9.x86_64 24/31 2023-11-08T12:24:27.181 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-common-2:18.0.0-7227.g25e18238.el9.x8 25/31 2023-11-08T12:24:29.305 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:18.0.0-7227.g25e18238.el9.x86_64 26/31 2023-11-08T12:24:29.344 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-common-2:18.0.0-7227.g25e18238.el9.x86_64 26/31 2023-11-08T12:24:30.119 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:18.0.0-7227.g25e18238.el9.x86_64 26/31 2023-11-08T12:24:30.165 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-base-2:18.0.0-7227.g25e18238.el9.x86_64 27/31 2023-11-08T12:24:30.247 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-base-2:18.0.0-7227.g25e18238.el9.x86_64 27/31 2023-11-08T12:24:30.247 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-11-08T12:24:30.247 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-11-08T12:24:30.248 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:24:30.283 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-selinux-2:18.0.0-7227.g25e18238.el9.x86_64 28/31 2023-11-08T12:24:40.197 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7227.g25e18238.el9.x86_64 28/31 2023-11-08T12:24:40.197 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /sys 2023-11-08T12:24:40.198 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /proc 2023-11-08T12:24:40.198 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /mnt 2023-11-08T12:24:40.198 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /var/tmp 2023-11-08T12:24:40.198 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /home 2023-11-08T12:24:40.199 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /root 2023-11-08T12:24:40.199 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /tmp 2023-11-08T12:24:40.199 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:24:41.071 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-radosgw-2:18.0.0-7227.g25e18238.el9.x86_64 29/31 2023-11-08T12:24:41.098 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7227.g25e18238.el9.x86_64 29/31 2023-11-08T12:24:41.098 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-08T12:24:41.098 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-11-08T12:24:41.099 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-08T12:24:41.099 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-08T12:24:41.099 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:24:41.101 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-11-08T12:24:41.182 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-11-08T12:24:41.183 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-11-08T12:24:42.336 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-11-08T12:24:42.336 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-base-2:18.0.0-7227.g25e18238.el9.x86_64 1/31 2023-11-08T12:24:42.337 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-common-2:18.0.0-7227.g25e18238.el9.x86_64 2/31 2023-11-08T12:24:42.337 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-radosgw-2:18.0.0-7227.g25e18238.el9.x86_64 3/31 2023-11-08T12:24:42.337 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-selinux-2:18.0.0-7227.g25e18238.el9.x86_64 4/31 2023-11-08T12:24:42.338 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs2-2:18.0.0-7227.g25e18238.el9.x86_64 5/31 2023-11-08T12:24:42.338 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libradosstriper1-2:18.0.0-7227.g25e18238.el9.x86_6 6/31 2023-11-08T12:24:42.338 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librgw2-2:18.0.0-7227.g25e18238.el9.x86_64 7/31 2023-11-08T12:24:42.338 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7227.g25e18238.el9. 8/31 2023-11-08T12:24:42.339 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-common-2:18.0.0-7227.g25e18238.el9.x8 9/31 2023-11-08T12:24:42.339 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cephfs-2:18.0.0-7227.g25e18238.el9.x86_64 10/31 2023-11-08T12:24:42.339 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rados-2:18.0.0-7227.g25e18238.el9.x86_64 11/31 2023-11-08T12:24:42.339 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rbd-2:18.0.0-7227.g25e18238.el9.x86_64 12/31 2023-11-08T12:24:42.340 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rgw-2:18.0.0-7227.g25e18238.el9.x86_64 13/31 2023-11-08T12:24:42.340 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-11-08T12:24:42.340 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-11-08T12:24:42.340 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/31 2023-11-08T12:24:42.340 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/31 2023-11-08T12:24:42.341 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/31 2023-11-08T12:24:42.341 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/31 2023-11-08T12:24:42.341 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-11-08T12:24:42.341 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 21/31 2023-11-08T12:24:42.342 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 22/31 2023-11-08T12:24:42.342 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-11-08T12:24:42.342 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-11-08T12:24:42.342 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 25/31 2023-11-08T12:24:42.343 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-11-08T12:24:42.343 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-11-08T12:24:42.343 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-2:18.0.0-7227.g25e18238.el9.x86_64 28/31 2023-11-08T12:24:42.343 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-11-08T12:24:42.343 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-2:18.0.0-7227.g25e18238.el9.x86_64 30/31 2023-11-08T12:24:43.316 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-11-08T12:24:43.317 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:24:43.317 INFO:teuthology.orchestra.run.smithi039.stdout:Upgraded: 2023-11-08T12:24:43.317 INFO:teuthology.orchestra.run.smithi039.stdout: librados2-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:24:43.317 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:24:43.318 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-11-08T12:24:43.318 INFO:teuthology.orchestra.run.smithi039.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-11-08T12:24:43.318 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:24:43.318 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:24:43.318 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:24:43.319 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:24:43.319 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-11-08T12:24:43.319 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-9.0.0-9.el9.x86_64 2023-11-08T12:24:43.319 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-11-08T12:24:43.320 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:24:43.320 INFO:teuthology.orchestra.run.smithi039.stdout: liboath-2.6.7-2.el9.x86_64 2023-11-08T12:24:43.320 INFO:teuthology.orchestra.run.smithi039.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-11-08T12:24:43.320 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-11-08T12:24:43.320 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:24:43.321 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-11-08T12:24:43.321 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:24:43.321 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind-1.6.2-1.el9.x86_64 2023-11-08T12:24:43.321 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-11-08T12:24:43.321 INFO:teuthology.orchestra.run.smithi039.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-11-08T12:24:43.322 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:24:43.322 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:24:43.322 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:24:43.322 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-11-08T12:24:43.322 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:24:43.323 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:24:43.323 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:24:43.323 INFO:teuthology.orchestra.run.smithi039.stdout: re2-1:20211101-3.el9.x86_64 2023-11-08T12:24:43.323 INFO:teuthology.orchestra.run.smithi039.stdout: thrift-0.15.0-2.el9.x86_64 2023-11-08T12:24:43.323 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:24:43.324 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:24:43.675 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-test 2023-11-08T12:24:44.270 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:29 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:24:44.428 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:24:44.429 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:24:44.430 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-11-08T12:24:44.430 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:24:44.430 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-11-08T12:24:44.430 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test x86_64 2:18.0.0-7227.g25e18238.el9 ceph 117 M 2023-11-08T12:24:44.431 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-11-08T12:24:44.431 INFO:teuthology.orchestra.run.smithi039.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2023-11-08T12:24:44.431 INFO:teuthology.orchestra.run.smithi039.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-11-08T12:24:44.431 INFO:teuthology.orchestra.run.smithi039.stdout: libasan x86_64 11.4.1-2.3.el9 CentOS-AppStream 418 k 2023-11-08T12:24:44.431 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite x86_64 2:18.0.0-7227.g25e18238.el9 ceph 158 k 2023-11-08T12:24:44.432 INFO:teuthology.orchestra.run.smithi039.stdout: libubsan x86_64 11.4.1-2.3.el9 CentOS-AppStream 186 k 2023-11-08T12:24:44.432 INFO:teuthology.orchestra.run.smithi039.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-11-08T12:24:44.432 INFO:teuthology.orchestra.run.smithi039.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-11-08T12:24:44.432 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-11-08T12:24:44.432 INFO:teuthology.orchestra.run.smithi039.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2023-11-08T12:24:44.433 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:24:44.433 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-11-08T12:24:44.433 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:24:44.433 INFO:teuthology.orchestra.run.smithi039.stdout:Install 10 Packages 2023-11-08T12:24:44.433 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:24:44.434 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 120 M 2023-11-08T12:24:44.434 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 1.1 G 2023-11-08T12:24:44.435 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-11-08T12:24:44.754 INFO:teuthology.orchestra.run.smithi039.stdout:(1/10): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 186 kB 00:00 2023-11-08T12:24:44.804 INFO:teuthology.orchestra.run.smithi039.stdout:(2/10): libasan-11.4.1-2.3.el9.x86_64.rpm 8.4 MB/s | 418 kB 00:00 2023-11-08T12:24:44.838 INFO:teuthology.orchestra.run.smithi039.stdout:(3/10): libubsan-11.4.1-2.3.el9.x86_64.rpm 5.5 MB/s | 186 kB 00:00 2023-11-08T12:24:44.872 INFO:teuthology.orchestra.run.smithi039.stdout:(4/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 6.4 MB/s | 217 kB 00:00 2023-11-08T12:24:44.914 INFO:teuthology.orchestra.run.smithi039.stdout:(5/10): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-11-08T12:24:44.947 INFO:teuthology.orchestra.run.smithi039.stdout:(6/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-11-08T12:24:45.006 INFO:teuthology.orchestra.run.smithi039.stdout:(7/10): libcephsqlite-18.0.0-7227.g25e18238.el9 398 kB/s | 158 kB 00:00 2023-11-08T12:24:45.206 INFO:teuthology.orchestra.run.smithi039.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 609 kB/s | 122 kB 00:00 2023-11-08T12:24:45.265 INFO:teuthology.orchestra.run.smithi039.stdout:(9/10): cryptopp-8.6.0-1.el9.x86_64.rpm 4.0 MB/s | 1.3 MB 00:00 2023-11-08T12:24:51.506 INFO:teuthology.orchestra.run.smithi039.stdout:(10/10): ceph-test-18.0.0-7227.g25e18238.el9.x8 17 MB/s | 117 MB 00:06 2023-11-08T12:24:51.509 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:24:51.509 INFO:teuthology.orchestra.run.smithi039.stdout:Total 17 MB/s | 120 MB 00:07 2023-11-08T12:24:51.510 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-11-08T12:24:51.533 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-11-08T12:24:51.534 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-11-08T12:24:52.408 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-11-08T12:24:52.409 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-11-08T12:24:53.680 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-11-08T12:24:53.755 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2023-11-08T12:24:53.790 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2023-11-08T12:24:53.834 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2023-11-08T12:24:53.869 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2023-11-08T12:24:53.898 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-11-08T12:24:53.939 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : jq-1.6-16.el9.x86_64 6/10 2023-11-08T12:24:53.976 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libubsan-11.4.1-2.3.el9.x86_64 7/10 2023-11-08T12:24:54.025 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libasan-11.4.1-2.3.el9.x86_64 8/10 2023-11-08T12:24:54.052 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephsqlite-2:18.0.0-7227.g25e18238.el9.x86_64 9/10 2023-11-08T12:25:11.266 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7227.g25e18238.el9.x86_64 9/10 2023-11-08T12:25:11.344 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-test-2:18.0.0-7227.g25e18238.el9.x86_64 10/10 2023-11-08T12:25:19.389 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-test-2:18.0.0-7227.g25e18238.el9.x86_64 10/10 2023-11-08T12:25:19.389 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-test-2:18.0.0-7227.g25e18238.el9.x86_64 1/10 2023-11-08T12:25:19.390 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephsqlite-2:18.0.0-7227.g25e18238.el9.x86_64 2/10 2023-11-08T12:25:19.390 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : jq-1.6-16.el9.x86_64 3/10 2023-11-08T12:25:19.390 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libasan-11.4.1-2.3.el9.x86_64 4/10 2023-11-08T12:25:19.390 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libubsan-11.4.1-2.3.el9.x86_64 5/10 2023-11-08T12:25:19.390 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/10 2023-11-08T12:25:19.391 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 7/10 2023-11-08T12:25:19.391 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/10 2023-11-08T12:25:19.391 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2023-11-08T12:25:19.834 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2023-11-08T12:25:19.835 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:25:19.835 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-11-08T12:25:19.835 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:25:19.836 INFO:teuthology.orchestra.run.smithi039.stdout: cryptopp-8.6.0-1.el9.x86_64 2023-11-08T12:25:19.836 INFO:teuthology.orchestra.run.smithi039.stdout: jq-1.6-16.el9.x86_64 2023-11-08T12:25:19.836 INFO:teuthology.orchestra.run.smithi039.stdout: libasan-11.4.1-2.3.el9.x86_64 2023-11-08T12:25:19.836 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:25:19.837 INFO:teuthology.orchestra.run.smithi039.stdout: libubsan-11.4.1-2.3.el9.x86_64 2023-11-08T12:25:19.837 INFO:teuthology.orchestra.run.smithi039.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-11-08T12:25:19.837 INFO:teuthology.orchestra.run.smithi039.stdout: socat-1.7.4.1-5.el9.x86_64 2023-11-08T12:25:19.838 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-11-08T12:25:19.838 INFO:teuthology.orchestra.run.smithi039.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2023-11-08T12:25:19.838 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:25:19.839 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:25:20.137 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph 2023-11-08T12:25:20.723 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:05 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:25:20.877 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:25:20.880 INFO:teuthology.orchestra.run.smithi039.stdout:========================================================================================= 2023-11-08T12:25:20.880 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-11-08T12:25:20.880 INFO:teuthology.orchestra.run.smithi039.stdout:========================================================================================= 2023-11-08T12:25:20.881 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-11-08T12:25:20.881 INFO:teuthology.orchestra.run.smithi039.stdout: ceph x86_64 2:18.0.0-7227.g25e18238.el9 ceph 6.5 k 2023-11-08T12:25:20.882 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-11-08T12:25:20.882 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds x86_64 2:18.0.0-7227.g25e18238.el9 ceph 2.0 M 2023-11-08T12:25:20.882 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr x86_64 2:18.0.0-7227.g25e18238.el9 ceph 1.5 M 2023-11-08T12:25:20.883 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7227.g25e18238.el9 ceph-noarch 243 k 2023-11-08T12:25:20.883 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon x86_64 2:18.0.0-7227.g25e18238.el9 ceph 8.9 M 2023-11-08T12:25:20.884 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd x86_64 2:18.0.0-7227.g25e18238.el9 ceph 117 M 2023-11-08T12:25:20.884 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-11-08T12:25:20.884 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-11-08T12:25:20.885 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-11-08T12:25:20.885 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-11-08T12:25:20.886 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-11-08T12:25:20.887 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-11-08T12:25:20.887 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-11-08T12:25:20.887 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-11-08T12:25:20.888 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-11-08T12:25:20.888 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-11-08T12:25:20.888 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-11-08T12:25:20.889 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-11-08T12:25:20.889 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-11-08T12:25:20.890 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-11-08T12:25:20.890 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-11-08T12:25:20.890 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-11-08T12:25:20.891 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-11-08T12:25:20.892 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-11-08T12:25:20.892 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-11-08T12:25:20.892 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-11-08T12:25:20.893 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-11-08T12:25:20.893 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-11-08T12:25:20.893 INFO:teuthology.orchestra.run.smithi039.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-11-08T12:25:20.893 INFO:teuthology.orchestra.run.smithi039.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-11-08T12:25:20.893 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-11-08T12:25:20.894 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-11-08T12:25:20.894 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-11-08T12:25:20.894 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:25:20.894 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-11-08T12:25:20.894 INFO:teuthology.orchestra.run.smithi039.stdout:========================================================================================= 2023-11-08T12:25:20.895 INFO:teuthology.orchestra.run.smithi039.stdout:Install 33 Packages 2023-11-08T12:25:20.895 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:25:20.896 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 133 M 2023-11-08T12:25:20.896 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 1.4 G 2023-11-08T12:25:20.896 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-11-08T12:25:21.534 INFO:teuthology.orchestra.run.smithi039.stdout:(1/33): ceph-18.0.0-7227.g25e18238.el9.x86_64.r 38 kB/s | 6.5 kB 00:00 2023-11-08T12:25:21.826 INFO:teuthology.orchestra.run.smithi039.stdout:(2/33): ceph-mgr-18.0.0-7227.g25e18238.el9.x86_ 3.1 MB/s | 1.5 MB 00:00 2023-11-08T12:25:21.909 INFO:teuthology.orchestra.run.smithi039.stdout:(3/33): ceph-mds-18.0.0-7227.g25e18238.el9.x86_ 3.6 MB/s | 2.0 MB 00:00 2023-11-08T12:25:21.985 INFO:teuthology.orchestra.run.smithi039.stdout:(4/33): ceph-mgr-modules-core-18.0.0-7227.g25e1 3.2 MB/s | 243 kB 00:00 2023-11-08T12:25:22.085 INFO:teuthology.orchestra.run.smithi039.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 144 kB 00:00 2023-11-08T12:25:22.110 INFO:teuthology.orchestra.run.smithi039.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2023-11-08T12:25:22.135 INFO:teuthology.orchestra.run.smithi039.stdout:(7/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2023-11-08T12:25:22.177 INFO:teuthology.orchestra.run.smithi039.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2023-11-08T12:25:22.211 INFO:teuthology.orchestra.run.smithi039.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-11-08T12:25:22.244 INFO:teuthology.orchestra.run.smithi039.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2023-11-08T12:25:22.278 INFO:teuthology.orchestra.run.smithi039.stdout:(11/33): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-11-08T12:25:22.320 INFO:teuthology.orchestra.run.smithi039.stdout:(12/33): python3-urllib3-1.26.5-4.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2023-11-08T12:25:22.562 INFO:teuthology.orchestra.run.smithi039.stdout:(13/33): ceph-mon-18.0.0-7227.g25e18238.el9.x86 8.6 MB/s | 8.9 MB 00:01 2023-11-08T12:25:23.029 INFO:teuthology.orchestra.run.smithi039.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 109 kB/s | 51 kB 00:00 2023-11-08T12:25:23.198 INFO:teuthology.orchestra.run.smithi039.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 261 kB/s | 43 kB 00:00 2023-11-08T12:25:23.271 INFO:teuthology.orchestra.run.smithi039.stdout:(16/33): python3-cryptography-36.0.1-4.el9.x86_ 1.3 MB/s | 1.2 MB 00:00 2023-11-08T12:25:23.338 INFO:teuthology.orchestra.run.smithi039.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2023-11-08T12:25:23.389 INFO:teuthology.orchestra.run.smithi039.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2023-11-08T12:25:23.447 INFO:teuthology.orchestra.run.smithi039.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 304 kB/s | 18 kB 00:00 2023-11-08T12:25:23.506 INFO:teuthology.orchestra.run.smithi039.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 398 kB/s | 23 kB 00:00 2023-11-08T12:25:23.531 INFO:teuthology.orchestra.run.smithi039.stdout:(21/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2023-11-08T12:25:23.565 INFO:teuthology.orchestra.run.smithi039.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 332 kB/s | 19 kB 00:00 2023-11-08T12:25:23.590 INFO:teuthology.orchestra.run.smithi039.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 337 kB/s | 20 kB 00:00 2023-11-08T12:25:23.615 INFO:teuthology.orchestra.run.smithi039.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 919 kB/s | 46 kB 00:00 2023-11-08T12:25:23.649 INFO:teuthology.orchestra.run.smithi039.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2023-11-08T12:25:23.699 INFO:teuthology.orchestra.run.smithi039.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2023-11-08T12:25:23.724 INFO:teuthology.orchestra.run.smithi039.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2023-11-08T12:25:23.775 INFO:teuthology.orchestra.run.smithi039.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2023-11-08T12:25:23.800 INFO:teuthology.orchestra.run.smithi039.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 475 kB/s | 36 kB 00:00 2023-11-08T12:25:23.859 INFO:teuthology.orchestra.run.smithi039.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2023-11-08T12:25:23.898 INFO:teuthology.orchestra.run.smithi039.stdout:(31/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.3 MB/s | 427 kB 00:00 2023-11-08T12:25:23.926 INFO:teuthology.orchestra.run.smithi039.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 300 kB/s | 20 kB 00:00 2023-11-08T12:25:27.376 INFO:teuthology.orchestra.run.smithi039.stdout:(33/33): ceph-osd-18.0.0-7227.g25e18238.el9.x86 21 MB/s | 117 MB 00:05 2023-11-08T12:25:27.380 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:25:27.380 INFO:teuthology.orchestra.run.smithi039.stdout:Total 21 MB/s | 133 MB 00:06 2023-11-08T12:25:27.414 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-11-08T12:25:27.458 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-11-08T12:25:27.458 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-11-08T12:25:28.475 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-11-08T12:25:28.475 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-11-08T12:25:30.353 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-11-08T12:25:30.420 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-11-08T12:25:30.453 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-11-08T12:25:30.485 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-11-08T12:25:30.588 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-11-08T12:25:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-11-08T12:25:30.713 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-11-08T12:25:30.751 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-11-08T12:25:30.830 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-11-08T12:25:30.881 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 9/33 2023-11-08T12:25:30.928 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-11-08T12:25:30.978 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-11-08T12:25:31.042 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-11-08T12:25:31.271 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-11-08T12:25:31.324 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-11-08T12:25:31.372 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-11-08T12:25:31.406 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-11-08T12:25:31.435 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-11-08T12:25:31.515 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-11-08T12:25:31.554 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-11-08T12:25:31.587 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-11-08T12:25:31.629 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-11-08T12:25:31.663 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-11-08T12:25:31.755 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-11-08T12:25:31.802 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-11-08T12:25:31.886 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-11-08T12:25:32.028 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-11-08T12:25:32.150 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-11-08T12:25:32.321 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7227.g25e18238.el9. 28/33 2023-11-08T12:25:32.357 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-2:18.0.0-7227.g25e18238.el9.x86_64 29/33 2023-11-08T12:25:32.383 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7227.g25e18238.el9.x86_64 29/33 2023-11-08T12:25:32.383 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-08T12:25:32.383 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-11-08T12:25:32.384 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-08T12:25:32.384 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-08T12:25:32.384 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:25:52.006 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-osd-2:18.0.0-7227.g25e18238.el9.x86_64 30/33 2023-11-08T12:25:52.034 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-osd-2:18.0.0-7227.g25e18238.el9.x86_64 30/33 2023-11-08T12:25:52.034 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-08T12:25:52.035 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-11-08T12:25:52.035 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-08T12:25:52.035 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-08T12:25:52.036 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:25:53.015 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mon-2:18.0.0-7227.g25e18238.el9.x86_64 31/33 2023-11-08T12:25:53.042 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mon-2:18.0.0-7227.g25e18238.el9.x86_64 31/33 2023-11-08T12:25:53.042 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-08T12:25:53.043 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-11-08T12:25:53.043 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-08T12:25:53.043 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-08T12:25:53.043 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:25:53.257 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mds-2:18.0.0-7227.g25e18238.el9.x86_64 32/33 2023-11-08T12:25:53.281 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mds-2:18.0.0-7227.g25e18238.el9.x86_64 32/33 2023-11-08T12:25:53.282 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-08T12:25:53.282 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-11-08T12:25:53.282 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-08T12:25:53.283 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-08T12:25:53.283 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:25:53.349 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-2:18.0.0-7227.g25e18238.el9.x86_64 33/33 2023-11-08T12:26:03.536 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-2:18.0.0-7227.g25e18238.el9.x86_64 33/33 2023-11-08T12:26:03.537 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-2:18.0.0-7227.g25e18238.el9.x86_64 1/33 2023-11-08T12:26:03.537 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mds-2:18.0.0-7227.g25e18238.el9.x86_64 2/33 2023-11-08T12:26:03.537 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-2:18.0.0-7227.g25e18238.el9.x86_64 3/33 2023-11-08T12:26:03.537 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mon-2:18.0.0-7227.g25e18238.el9.x86_64 4/33 2023-11-08T12:26:03.538 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-osd-2:18.0.0-7227.g25e18238.el9.x86_64 5/33 2023-11-08T12:26:03.538 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7227.g25e18238.el9. 6/33 2023-11-08T12:26:03.538 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-11-08T12:26:03.538 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-11-08T12:26:03.538 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-11-08T12:26:03.539 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-11-08T12:26:03.539 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-11-08T12:26:03.539 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-11-08T12:26:03.539 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-11-08T12:26:03.539 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 14/33 2023-11-08T12:26:03.540 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-11-08T12:26:03.540 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-11-08T12:26:03.540 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-11-08T12:26:03.540 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-11-08T12:26:03.540 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-11-08T12:26:03.541 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-11-08T12:26:03.541 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-11-08T12:26:03.541 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-11-08T12:26:03.541 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-11-08T12:26:03.542 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-11-08T12:26:03.542 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-11-08T12:26:03.542 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-11-08T12:26:03.542 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-11-08T12:26:03.543 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-11-08T12:26:03.543 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-11-08T12:26:03.544 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-11-08T12:26:03.544 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-11-08T12:26:03.544 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-11-08T12:26:04.495 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-11-08T12:26:04.495 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:04.495 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-11-08T12:26:04.495 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:26:04.496 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:26:04.496 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:26:04.496 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core-2:18.0.0-7227.g25e18238.el9.noarch 2023-11-08T12:26:04.496 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:26:04.496 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:26:04.497 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-11-08T12:26:04.497 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-11-08T12:26:04.497 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-11-08T12:26:04.497 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-11-08T12:26:04.497 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-11-08T12:26:04.498 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-11-08T12:26:04.498 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-11-08T12:26:04.498 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-11-08T12:26:04.498 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-11-08T12:26:04.498 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-11-08T12:26:04.499 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-11-08T12:26:04.499 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako-1.1.4-6.el9.noarch 2023-11-08T12:26:04.499 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-11-08T12:26:04.499 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-11-08T12:26:04.499 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-11-08T12:26:04.500 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ply-3.11-14.el9.noarch 2023-11-08T12:26:04.500 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend-3.1.0-2.el9.noarch 2023-11-08T12:26:04.500 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-11-08T12:26:04.500 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pycparser-2.20-6.el9.noarch 2023-11-08T12:26:04.500 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz-2021.1-5.el9.noarch 2023-11-08T12:26:04.500 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-2.25.1-7.el9.noarch 2023-11-08T12:26:04.501 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-11-08T12:26:04.501 INFO:teuthology.orchestra.run.smithi039.stdout: python3-toml-0.10.2-6.el9.noarch 2023-11-08T12:26:04.501 INFO:teuthology.orchestra.run.smithi039.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-11-08T12:26:04.501 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob-1.8.7-6.el9.noarch 2023-11-08T12:26:04.501 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-11-08T12:26:04.502 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-11-08T12:26:04.502 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:04.502 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:26:04.731 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-base 2023-11-08T12:26:05.322 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:50 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:26:05.419 INFO:teuthology.orchestra.run.smithi039.stdout:Package ceph-base-2:18.0.0-7227.g25e18238.el9.x86_64 is already installed. 2023-11-08T12:26:05.471 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:26:05.472 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-11-08T12:26:05.472 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:26:05.541 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install cephadm 2023-11-08T12:26:06.128 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:51 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:26:06.278 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:26:06.279 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:06.279 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-11-08T12:26:06.280 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:06.280 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-11-08T12:26:06.280 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm noarch 2:18.0.0-7227.g25e18238.el9 ceph-noarch 768 k 2023-11-08T12:26:06.281 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:06.281 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-11-08T12:26:06.281 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:06.282 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2023-11-08T12:26:06.282 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:06.282 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 768 k 2023-11-08T12:26:06.283 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 782 k 2023-11-08T12:26:06.283 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-11-08T12:26:06.548 INFO:teuthology.orchestra.run.smithi039.stdout:cephadm-18.0.0-7227.g25e18238.el9.noarch.rpm 2.8 MB/s | 768 kB 00:00 2023-11-08T12:26:06.548 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:26:06.549 INFO:teuthology.orchestra.run.smithi039.stdout:Total 2.8 MB/s | 768 kB 00:00 2023-11-08T12:26:06.549 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-11-08T12:26:06.554 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-11-08T12:26:06.555 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-11-08T12:26:06.560 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-11-08T12:26:06.561 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-11-08T12:26:06.637 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-11-08T12:26:07.127 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephadm-2:18.0.0-7227.g25e18238.el9.noarch 1/1 2023-11-08T12:26:07.221 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephadm-2:18.0.0-7227.g25e18238.el9.noarch 1/1 2023-11-08T12:26:07.599 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephadm-2:18.0.0-7227.g25e18238.el9.noarch 1/1 2023-11-08T12:26:07.919 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephadm-2:18.0.0-7227.g25e18238.el9.noarch 1/1 2023-11-08T12:26:07.920 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:07.920 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-11-08T12:26:07.920 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm-2:18.0.0-7227.g25e18238.el9.noarch 2023-11-08T12:26:07.920 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:07.920 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:26:08.140 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-immutable-object-cache 2023-11-08T12:26:08.743 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:53 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:26:08.892 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:26:08.893 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:08.893 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2023-11-08T12:26:08.893 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:08.894 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-11-08T12:26:08.894 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7227.g25e18238.el9 ceph 143 k 2023-11-08T12:26:08.894 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:08.894 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-11-08T12:26:08.894 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:08.895 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2023-11-08T12:26:08.895 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:08.895 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 143 k 2023-11-08T12:26:08.895 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 438 k 2023-11-08T12:26:08.895 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-11-08T12:26:09.195 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-immutable-object-cache-18.0.0-7227.g25e182 476 kB/s | 143 kB 00:00 2023-11-08T12:26:09.197 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:26:09.197 INFO:teuthology.orchestra.run.smithi039.stdout:Total 474 kB/s | 143 kB 00:00 2023-11-08T12:26:09.198 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-11-08T12:26:09.204 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-11-08T12:26:09.204 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-11-08T12:26:09.259 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-11-08T12:26:09.260 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-11-08T12:26:09.420 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-11-08T12:26:09.479 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7227.g25e18238. 1/1 2023-11-08T12:26:09.505 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7227.g25e18238. 1/1 2023-11-08T12:26:09.506 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-08T12:26:09.506 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-11-08T12:26:09.506 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:10.517 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7227.g25e18238. 1/1 2023-11-08T12:26:10.517 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:10.517 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-11-08T12:26:10.517 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:26:10.518 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:10.518 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:26:10.766 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr 2023-11-08T12:26:11.359 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:56 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:26:11.456 INFO:teuthology.orchestra.run.smithi039.stdout:Package ceph-mgr-2:18.0.0-7227.g25e18238.el9.x86_64 is already installed. 2023-11-08T12:26:11.508 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:26:11.509 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-11-08T12:26:11.510 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:26:11.586 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-dashboard 2023-11-08T12:26:12.185 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:57 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:26:12.337 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:26:12.338 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:12.338 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-11-08T12:26:12.338 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:12.339 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-11-08T12:26:12.339 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7227.g25e18238.el9 ceph-noarch 1.8 M 2023-11-08T12:26:12.339 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-11-08T12:26:12.339 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7227.g25e18238.el9 ceph-noarch 23 k 2023-11-08T12:26:12.339 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7227.g25e18238.el9 ceph-noarch 15 k 2023-11-08T12:26:12.340 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-11-08T12:26:12.340 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-11-08T12:26:12.340 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-11-08T12:26:12.340 INFO:teuthology.orchestra.run.smithi039.stdout:Installing weak dependencies: 2023-11-08T12:26:12.340 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-11-08T12:26:12.341 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:12.341 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-11-08T12:26:12.341 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:12.341 INFO:teuthology.orchestra.run.smithi039.stdout:Install 7 Packages 2023-11-08T12:26:12.341 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:12.342 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 2.1 M 2023-11-08T12:26:12.342 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 8.6 M 2023-11-08T12:26:12.342 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-11-08T12:26:12.682 INFO:teuthology.orchestra.run.smithi039.stdout:(1/7): ceph-grafana-dashboards-18.0.0-7227.g25e 89 kB/s | 23 kB 00:00 2023-11-08T12:26:12.708 INFO:teuthology.orchestra.run.smithi039.stdout:(2/7): ceph-prometheus-alerts-18.0.0-7227.g25e1 53 kB/s | 15 kB 00:00 2023-11-08T12:26:12.949 INFO:teuthology.orchestra.run.smithi039.stdout:(3/7): ceph-mgr-dashboard-18.0.0-7227.g25e18238 3.3 MB/s | 1.8 MB 00:00 2023-11-08T12:26:13.000 INFO:teuthology.orchestra.run.smithi039.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 28 kB/s | 9.0 kB 00:00 2023-11-08T12:26:13.175 INFO:teuthology.orchestra.run.smithi039.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 87 kB/s | 41 kB 00:00 2023-11-08T12:26:13.334 INFO:teuthology.orchestra.run.smithi039.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 80 kB/s | 31 kB 00:00 2023-11-08T12:26:13.359 INFO:teuthology.orchestra.run.smithi039.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 525 kB/s | 188 kB 00:00 2023-11-08T12:26:13.362 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:26:13.363 INFO:teuthology.orchestra.run.smithi039.stdout:Total 2.0 MB/s | 2.1 MB 00:01 2023-11-08T12:26:13.363 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-11-08T12:26:13.371 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-11-08T12:26:13.372 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-11-08T12:26:13.445 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-11-08T12:26:13.445 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-11-08T12:26:13.869 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-11-08T12:26:13.927 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-11-08T12:26:13.962 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-11-08T12:26:14.009 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-11-08T12:26:14.048 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-11-08T12:26:14.077 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7227.g25e18238.el9.n 5/7 2023-11-08T12:26:14.325 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7227.g25e18238.el9. 6/7 2023-11-08T12:26:14.354 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7227.g25e18238.el9.noarc 7/7 2023-11-08T12:26:14.896 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7227.g25e18238.el9.noarc 7/7 2023-11-08T12:26:14.896 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7227.g25e18238.el9. 1/7 2023-11-08T12:26:14.897 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7227.g25e18238.el9.noarc 2/7 2023-11-08T12:26:14.897 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7227.g25e18238.el9.n 3/7 2023-11-08T12:26:14.897 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-11-08T12:26:14.897 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-11-08T12:26:14.898 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-11-08T12:26:15.203 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-11-08T12:26:15.203 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:15.203 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-11-08T12:26:15.203 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards-2:18.0.0-7227.g25e18238.el9.noarch 2023-11-08T12:26:15.204 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard-2:18.0.0-7227.g25e18238.el9.noarch 2023-11-08T12:26:15.204 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts-2:18.0.0-7227.g25e18238.el9.noarch 2023-11-08T12:26:15.204 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-11-08T12:26:15.204 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-11-08T12:26:15.204 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-11-08T12:26:15.205 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes-2.5.1-5.el9.noarch 2023-11-08T12:26:15.205 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:15.205 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:26:15.420 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-08T12:26:16.011 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:01 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:26:16.166 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:26:16.166 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================================ 2023-11-08T12:26:16.166 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-11-08T12:26:16.167 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================================ 2023-11-08T12:26:16.167 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-11-08T12:26:16.167 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7227.g25e18238.el9 ceph-noarch 7.4 M 2023-11-08T12:26:16.167 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-11-08T12:26:16.168 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-11-08T12:26:16.168 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-11-08T12:26:16.168 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-11-08T12:26:16.168 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-11-08T12:26:16.169 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-11-08T12:26:16.169 INFO:teuthology.orchestra.run.smithi039.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-11-08T12:26:16.169 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-11-08T12:26:16.169 INFO:teuthology.orchestra.run.smithi039.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-11-08T12:26:16.169 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-11-08T12:26:16.170 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-11-08T12:26:16.170 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-11-08T12:26:16.170 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:16.170 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-11-08T12:26:16.170 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================================ 2023-11-08T12:26:16.171 INFO:teuthology.orchestra.run.smithi039.stdout:Install 12 Packages 2023-11-08T12:26:16.171 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:16.171 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 37 M 2023-11-08T12:26:16.172 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 209 M 2023-11-08T12:26:16.172 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-11-08T12:26:16.290 INFO:teuthology.orchestra.run.smithi039.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 258 kB/s | 32 kB 00:00 2023-11-08T12:26:16.324 INFO:teuthology.orchestra.run.smithi039.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 533 kB/s | 17 kB 00:00 2023-11-08T12:26:16.358 INFO:teuthology.orchestra.run.smithi039.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-11-08T12:26:16.416 INFO:teuthology.orchestra.run.smithi039.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2023-11-08T12:26:16.475 INFO:teuthology.orchestra.run.smithi039.stdout:(5/12): python3-devel-3.9.17-2.el9.x86_64.rpm 3.5 MB/s | 205 kB 00:00 2023-11-08T12:26:16.542 INFO:teuthology.orchestra.run.smithi039.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2023-11-08T12:26:16.626 INFO:teuthology.orchestra.run.smithi039.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2023-11-08T12:26:16.651 INFO:teuthology.orchestra.run.smithi039.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 217 kB 00:00 2023-11-08T12:26:16.709 INFO:teuthology.orchestra.run.smithi039.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 14 MB/s | 803 kB 00:00 2023-11-08T12:26:16.893 INFO:teuthology.orchestra.run.smithi039.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 59 MB/s | 16 MB 00:00 2023-11-08T12:26:16.918 INFO:teuthology.orchestra.run.smithi039.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 947 kB/s | 195 kB 00:00 2023-11-08T12:26:17.035 INFO:teuthology.orchestra.run.smithi039.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-7 8.5 MB/s | 7.4 MB 00:00 2023-11-08T12:26:17.038 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:26:17.038 INFO:teuthology.orchestra.run.smithi039.stdout:Total 42 MB/s | 37 MB 00:00 2023-11-08T12:26:17.038 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-11-08T12:26:17.062 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-11-08T12:26:17.063 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-11-08T12:26:17.522 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-11-08T12:26:17.522 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-11-08T12:26:18.247 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-11-08T12:26:18.325 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-11-08T12:26:18.356 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-11-08T12:26:18.455 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-11-08T12:26:18.493 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-11-08T12:26:18.670 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-08T12:26:18.725 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-08T12:26:18.812 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-11-08T12:26:19.500 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-11-08T12:26:19.617 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-11-08T12:26:20.928 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-11-08T12:26:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-11-08T12:26:22.058 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7227.g25e18 12/12 2023-11-08T12:26:24.606 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7227.g25e18 12/12 2023-11-08T12:26:24.606 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7227.g25e18 1/12 2023-11-08T12:26:24.607 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-11-08T12:26:24.607 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-11-08T12:26:24.607 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-11-08T12:26:24.607 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-08T12:26:24.607 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-08T12:26:24.608 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-11-08T12:26:24.608 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-11-08T12:26:24.608 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-11-08T12:26:24.608 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-11-08T12:26:24.608 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-11-08T12:26:25.007 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-11-08T12:26:25.008 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:25.008 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-11-08T12:26:25.008 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7227.g25e18238.el9.noarch 2023-11-08T12:26:25.008 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-11-08T12:26:25.009 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-11-08T12:26:25.009 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-11-08T12:26:25.009 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-11-08T12:26:25.009 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-11-08T12:26:25.009 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-0.3.21-2.el9.x86_64 2023-11-08T12:26:25.010 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-11-08T12:26:25.010 INFO:teuthology.orchestra.run.smithi039.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-11-08T12:26:25.010 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-11-08T12:26:25.010 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-11-08T12:26:25.010 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-11-08T12:26:25.011 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:25.011 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:26:25.391 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-rook 2023-11-08T12:26:25.995 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:10 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:26:26.146 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:26:26.148 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================== 2023-11-08T12:26:26.148 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-11-08T12:26:26.148 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================== 2023-11-08T12:26:26.148 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-11-08T12:26:26.148 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook noarch 2:18.0.0-7227.g25e18238.el9 ceph-noarch 50 k 2023-11-08T12:26:26.149 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-11-08T12:26:26.149 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-11-08T12:26:26.149 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-11-08T12:26:26.149 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth noarch 1:2.23.3-1.el9 epel 197 k 2023-11-08T12:26:26.149 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-11-08T12:26:26.150 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-11-08T12:26:26.150 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-11-08T12:26:26.150 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-11-08T12:26:26.150 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-11-08T12:26:26.150 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-11-08T12:26:26.151 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-11-08T12:26:26.151 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-11-08T12:26:26.151 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-11-08T12:26:26.151 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:26.151 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-11-08T12:26:26.152 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================== 2023-11-08T12:26:26.152 INFO:teuthology.orchestra.run.smithi039.stdout:Install 13 Packages 2023-11-08T12:26:26.152 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:26.153 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 2.0 M 2023-11-08T12:26:26.153 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 26 M 2023-11-08T12:26:26.153 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-11-08T12:26:26.633 INFO:teuthology.orchestra.run.smithi039.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 194 kB/s | 24 kB 00:00 2023-11-08T12:26:26.658 INFO:teuthology.orchestra.run.smithi039.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 120 kB/s | 18 kB 00:00 2023-11-08T12:26:26.700 INFO:teuthology.orchestra.run.smithi039.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2023-11-08T12:26:26.725 INFO:teuthology.orchestra.run.smithi039.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-11-08T12:26:26.750 INFO:teuthology.orchestra.run.smithi039.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-11-08T12:26:26.775 INFO:teuthology.orchestra.run.smithi039.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 857 kB/s | 43 kB 00:00 2023-11-08T12:26:26.826 INFO:teuthology.orchestra.run.smithi039.stdout:(7/13): ceph-mgr-rook-18.0.0-7227.g25e18238.el9 157 kB/s | 50 kB 00:00 2023-11-08T12:26:26.951 INFO:teuthology.orchestra.run.smithi039.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 160 kB/s | 32 kB 00:00 2023-11-08T12:26:26.976 INFO:teuthology.orchestra.run.smithi039.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 75 kB/s | 15 kB 00:00 2023-11-08T12:26:27.093 INFO:teuthology.orchestra.run.smithi039.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 506 kB/s | 59 kB 00:00 2023-11-08T12:26:27.118 INFO:teuthology.orchestra.run.smithi039.stdout:(11/13): python3-google-auth-2.23.3-1.el9.noarc 674 kB/s | 197 kB 00:00 2023-11-08T12:26:27.169 INFO:teuthology.orchestra.run.smithi039.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.2 MB/s | 90 kB 00:00 2023-11-08T12:26:27.227 INFO:teuthology.orchestra.run.smithi039.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.7 MB/s | 1.0 MB 00:00 2023-11-08T12:26:27.229 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:26:27.229 INFO:teuthology.orchestra.run.smithi039.stdout:Total 1.9 MB/s | 2.0 MB 00:01 2023-11-08T12:26:27.230 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-11-08T12:26:27.244 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-11-08T12:26:27.244 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-11-08T12:26:27.350 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-11-08T12:26:27.350 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-11-08T12:26:27.886 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-11-08T12:26:28.050 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-11-08T12:26:28.113 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-11-08T12:26:28.169 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-11-08T12:26:28.205 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-11-08T12:26:28.247 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-11-08T12:26:28.343 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-11-08T12:26:28.453 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-google-auth-1:2.23.3-1.el9.noarch 7/13 2023-11-08T12:26:28.515 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-11-08T12:26:29.275 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-11-08T12:26:29.319 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-11-08T12:26:29.353 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-11-08T12:26:29.402 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-11-08T12:26:29.431 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-rook-2:18.0.0-7227.g25e18238.el9.noarch 13/13 2023-11-08T12:26:30.262 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7227.g25e18238.el9.noarch 13/13 2023-11-08T12:26:30.262 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7227.g25e18238.el9.noarch 1/13 2023-11-08T12:26:30.262 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-11-08T12:26:30.263 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-11-08T12:26:30.263 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-11-08T12:26:30.263 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-11-08T12:26:30.263 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-11-08T12:26:30.264 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-11-08T12:26:30.264 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-11-08T12:26:30.264 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-11-08T12:26:30.264 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-google-auth-1:2.23.3-1.el9.noarch 10/13 2023-11-08T12:26:30.264 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-11-08T12:26:30.265 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-11-08T12:26:30.635 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-11-08T12:26:30.635 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:30.635 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-11-08T12:26:30.636 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook-2:18.0.0-7227.g25e18238.el9.noarch 2023-11-08T12:26:30.636 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-11-08T12:26:30.636 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-11-08T12:26:30.636 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth-1:2.23.3-1.el9.noarch 2023-11-08T12:26:30.636 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-11-08T12:26:30.637 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-11-08T12:26:30.637 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-11-08T12:26:30.637 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-11-08T12:26:30.637 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-11-08T12:26:30.637 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-11-08T12:26:30.638 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-11-08T12:26:30.638 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa-4.9-2.el9.noarch 2023-11-08T12:26:30.638 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-11-08T12:26:30.638 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:30.638 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:26:30.883 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-mgr-cephadm 2023-11-08T12:26:31.483 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:16 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:26:31.637 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:26:31.637 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================== 2023-11-08T12:26:31.638 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-11-08T12:26:31.638 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================== 2023-11-08T12:26:31.638 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-11-08T12:26:31.638 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7227.g25e18238.el9 ceph-noarch 132 k 2023-11-08T12:26:31.639 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-11-08T12:26:31.639 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-11-08T12:26:31.639 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-11-08T12:26:31.639 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-11-08T12:26:31.639 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-11-08T12:26:31.640 INFO:teuthology.orchestra.run.smithi039.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-11-08T12:26:31.640 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:31.640 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-11-08T12:26:31.640 INFO:teuthology.orchestra.run.smithi039.stdout:======================================================================================== 2023-11-08T12:26:31.640 INFO:teuthology.orchestra.run.smithi039.stdout:Install 6 Packages 2023-11-08T12:26:31.641 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:31.641 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 6.8 M 2023-11-08T12:26:31.641 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 33 M 2023-11-08T12:26:31.642 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-11-08T12:26:32.162 INFO:teuthology.orchestra.run.smithi039.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2023-11-08T12:26:32.271 INFO:teuthology.orchestra.run.smithi039.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2023-11-08T12:26:32.338 INFO:teuthology.orchestra.run.smithi039.stdout:(3/6): ceph-mgr-cephadm-18.0.0-7227.g25e18238.e 399 kB/s | 132 kB 00:00 2023-11-08T12:26:32.589 INFO:teuthology.orchestra.run.smithi039.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 197 kB/s | 58 kB 00:00 2023-11-08T12:26:32.613 INFO:teuthology.orchestra.run.smithi039.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 185 kB/s | 51 kB 00:00 2023-11-08T12:26:32.713 INFO:teuthology.orchestra.run.smithi039.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 996 kB/s | 548 kB 00:00 2023-11-08T12:26:32.717 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:26:32.718 INFO:teuthology.orchestra.run.smithi039.stdout:Total 6.3 MB/s | 6.8 MB 00:01 2023-11-08T12:26:32.718 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-11-08T12:26:32.728 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-11-08T12:26:32.728 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-11-08T12:26:32.822 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-11-08T12:26:32.823 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-11-08T12:26:33.132 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-11-08T12:26:33.314 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-11-08T12:26:33.366 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-11-08T12:26:33.926 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-11-08T12:26:34.005 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-11-08T12:26:34.074 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-11-08T12:26:34.104 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7227.g25e18238.el9.noarch 6/6 2023-11-08T12:26:34.907 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7227.g25e18238.el9.noarch 6/6 2023-11-08T12:26:34.908 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7227.g25e18238.el9.noarch 1/6 2023-11-08T12:26:34.908 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-11-08T12:26:34.908 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-11-08T12:26:34.909 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-11-08T12:26:34.909 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-11-08T12:26:35.214 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-11-08T12:26:35.214 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:35.214 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-11-08T12:26:35.215 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm-2:18.0.0-7227.g25e18238.el9.noarch 2023-11-08T12:26:35.215 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-11-08T12:26:35.215 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel-2.9.1-2.el9.noarch 2023-11-08T12:26:35.215 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-11-08T12:26:35.215 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-11-08T12:26:35.216 INFO:teuthology.orchestra.run.smithi039.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-11-08T12:26:35.216 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:35.216 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:26:35.447 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-fuse 2023-11-08T12:26:36.039 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:21 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:26:36.189 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:26:36.190 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:36.191 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-11-08T12:26:36.191 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:36.191 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-11-08T12:26:36.191 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse x86_64 2:18.0.0-7227.g25e18238.el9 ceph 833 k 2023-11-08T12:26:36.192 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-11-08T12:26:36.192 INFO:teuthology.orchestra.run.smithi039.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-11-08T12:26:36.192 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:36.192 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-11-08T12:26:36.192 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:36.193 INFO:teuthology.orchestra.run.smithi039.stdout:Install 2 Packages 2023-11-08T12:26:36.193 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:36.193 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 911 k 2023-11-08T12:26:36.193 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 3.0 M 2023-11-08T12:26:36.193 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-11-08T12:26:36.292 INFO:teuthology.orchestra.run.smithi039.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 786 kB/s | 79 kB 00:00 2023-11-08T12:26:36.609 INFO:teuthology.orchestra.run.smithi039.stdout:(2/2): ceph-fuse-18.0.0-7227.g25e18238.el9.x86_ 1.9 MB/s | 833 kB 00:00 2023-11-08T12:26:36.610 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:26:36.610 INFO:teuthology.orchestra.run.smithi039.stdout:Total 2.1 MB/s | 911 kB 00:00 2023-11-08T12:26:36.611 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-11-08T12:26:36.631 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-11-08T12:26:36.632 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-11-08T12:26:36.711 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-11-08T12:26:36.711 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-11-08T12:26:36.902 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-11-08T12:26:37.063 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-11-08T12:26:37.136 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-fuse-2:18.0.0-7227.g25e18238.el9.x86_64 2/2 2023-11-08T12:26:37.864 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7227.g25e18238.el9.x86_64 2/2 2023-11-08T12:26:37.865 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-fuse-2:18.0.0-7227.g25e18238.el9.x86_64 1/2 2023-11-08T12:26:38.158 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-11-08T12:26:38.159 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:38.159 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-11-08T12:26:38.159 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse-2:18.0.0-7227.g25e18238.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-11-08T12:26:38.159 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:38.160 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:26:38.434 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-volume 2023-11-08T12:26:39.038 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:24 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:26:39.190 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:26:39.191 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:39.191 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-11-08T12:26:39.191 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:39.191 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-11-08T12:26:39.192 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-volume noarch 2:18.0.0-7227.g25e18238.el9 ceph-noarch 262 k 2023-11-08T12:26:39.192 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:39.192 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-11-08T12:26:39.192 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:39.192 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2023-11-08T12:26:39.193 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:39.193 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 262 k 2023-11-08T12:26:39.193 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 1.2 M 2023-11-08T12:26:39.193 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-11-08T12:26:39.538 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-volume-18.0.0-7227.g25e18238.el9.noarch.rp 757 kB/s | 262 kB 00:00 2023-11-08T12:26:39.539 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:26:39.539 INFO:teuthology.orchestra.run.smithi039.stdout:Total 753 kB/s | 262 kB 00:00 2023-11-08T12:26:39.540 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-11-08T12:26:39.548 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-11-08T12:26:39.548 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-11-08T12:26:39.580 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-11-08T12:26:39.581 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-11-08T12:26:39.807 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-11-08T12:26:39.865 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-volume-2:18.0.0-7227.g25e18238.el9.noarch 1/1 2023-11-08T12:26:39.881 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-volume-2:18.0.0-7227.g25e18238.el9.noarch 1/1 2023-11-08T12:26:39.881 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-08T12:26:39.882 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-11-08T12:26:39.882 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:40.929 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-volume-2:18.0.0-7227.g25e18238.el9.noarch 1/1 2023-11-08T12:26:40.929 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:40.930 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-11-08T12:26:40.930 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-volume-2:18.0.0-7227.g25e18238.el9.noarch 2023-11-08T12:26:40.930 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:40.930 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:26:41.216 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install librados-devel 2023-11-08T12:26:41.820 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:26 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:26:41.973 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:26:41.974 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:41.974 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2023-11-08T12:26:41.974 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:41.975 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-11-08T12:26:41.975 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel x86_64 2:18.0.0-7227.g25e18238.el9 ceph 127 k 2023-11-08T12:26:41.975 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:41.975 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-11-08T12:26:41.975 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:41.976 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2023-11-08T12:26:41.976 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:41.976 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 127 k 2023-11-08T12:26:41.976 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 460 k 2023-11-08T12:26:41.976 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-11-08T12:26:42.294 INFO:teuthology.orchestra.run.smithi039.stdout:librados-devel-18.0.0-7227.g25e18238.el9.x86_64 398 kB/s | 127 kB 00:00 2023-11-08T12:26:42.295 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:26:42.295 INFO:teuthology.orchestra.run.smithi039.stdout:Total 396 kB/s | 127 kB 00:00 2023-11-08T12:26:42.295 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-11-08T12:26:42.301 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-11-08T12:26:42.301 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-11-08T12:26:42.361 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-11-08T12:26:42.361 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-11-08T12:26:42.516 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-11-08T12:26:42.612 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librados-devel-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:26:43.094 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados-devel-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:26:43.396 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados-devel-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:26:43.397 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:43.397 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-11-08T12:26:43.397 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:26:43.397 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:43.397 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:26:43.658 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install libcephfs2 2023-11-08T12:26:44.265 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:29 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:26:44.363 INFO:teuthology.orchestra.run.smithi039.stdout:Package libcephfs2-2:18.0.0-7227.g25e18238.el9.x86_64 is already installed. 2023-11-08T12:26:44.416 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:26:44.417 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-11-08T12:26:44.417 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:26:44.490 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install libcephfs-devel 2023-11-08T12:26:45.087 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:30 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:26:45.240 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:26:45.241 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:45.241 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2023-11-08T12:26:45.241 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:45.241 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-11-08T12:26:45.241 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel x86_64 2:18.0.0-7227.g25e18238.el9 ceph 31 k 2023-11-08T12:26:45.242 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:45.242 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-11-08T12:26:45.242 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:45.242 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2023-11-08T12:26:45.243 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:45.243 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 31 k 2023-11-08T12:26:45.243 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 139 k 2023-11-08T12:26:45.243 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-11-08T12:26:45.489 INFO:teuthology.orchestra.run.smithi039.stdout:libcephfs-devel-18.0.0-7227.g25e18238.el9.x86_6 125 kB/s | 31 kB 00:00 2023-11-08T12:26:45.490 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:26:45.490 INFO:teuthology.orchestra.run.smithi039.stdout:Total 125 kB/s | 31 kB 00:00 2023-11-08T12:26:45.491 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-11-08T12:26:45.494 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-11-08T12:26:45.494 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-11-08T12:26:45.524 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-11-08T12:26:45.524 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-11-08T12:26:45.619 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-11-08T12:26:45.731 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs-devel-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:26:46.139 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:26:46.402 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs-devel-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:26:46.403 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:46.403 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-11-08T12:26:46.403 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:26:46.403 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:46.403 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:26:46.571 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install librados2 2023-11-08T12:26:47.165 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:32 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:26:47.263 INFO:teuthology.orchestra.run.smithi039.stdout:Package librados2-2:18.0.0-7227.g25e18238.el9.x86_64 is already installed. 2023-11-08T12:26:47.316 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:26:47.317 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-11-08T12:26:47.317 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:26:47.384 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install librbd1 2023-11-08T12:26:47.976 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:32 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:26:48.073 INFO:teuthology.orchestra.run.smithi039.stdout:Package librbd1-2:18.0.0-7227.g25e18238.el9.x86_64 is already installed. 2023-11-08T12:26:48.126 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:26:48.127 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-11-08T12:26:48.127 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:26:48.192 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-rados 2023-11-08T12:26:48.795 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:33 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:26:48.892 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-rados-2:18.0.0-7227.g25e18238.el9.x86_64 is already installed. 2023-11-08T12:26:48.945 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:26:48.946 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-11-08T12:26:48.946 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:26:49.017 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-rgw 2023-11-08T12:26:49.619 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:34 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:26:49.716 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-rgw-2:18.0.0-7227.g25e18238.el9.x86_64 is already installed. 2023-11-08T12:26:49.769 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:26:49.770 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-11-08T12:26:49.770 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:26:49.846 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-cephfs 2023-11-08T12:26:50.445 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:35 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:26:50.543 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-cephfs-2:18.0.0-7227.g25e18238.el9.x86_64 is already installed. 2023-11-08T12:26:50.596 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:26:50.598 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-11-08T12:26:50.598 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:26:50.667 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-rbd 2023-11-08T12:26:51.265 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:36 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:26:51.362 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-rbd-2:18.0.0-7227.g25e18238.el9.x86_64 is already installed. 2023-11-08T12:26:51.416 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:26:51.417 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-11-08T12:26:51.417 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:26:51.487 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install rbd-fuse 2023-11-08T12:26:52.079 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:37 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:26:52.231 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:26:52.231 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:52.232 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2023-11-08T12:26:52.232 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:52.232 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-11-08T12:26:52.232 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse x86_64 2:18.0.0-7227.g25e18238.el9 ceph 86 k 2023-11-08T12:26:52.233 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:52.233 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-11-08T12:26:52.233 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:52.233 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2023-11-08T12:26:52.233 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:52.234 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 86 k 2023-11-08T12:26:52.234 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 256 k 2023-11-08T12:26:52.234 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-11-08T12:26:52.542 INFO:teuthology.orchestra.run.smithi039.stdout:rbd-fuse-18.0.0-7227.g25e18238.el9.x86_64.rpm 277 kB/s | 86 kB 00:00 2023-11-08T12:26:52.543 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:26:52.543 INFO:teuthology.orchestra.run.smithi039.stdout:Total 276 kB/s | 86 kB 00:00 2023-11-08T12:26:52.544 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-11-08T12:26:52.550 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-11-08T12:26:52.551 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-11-08T12:26:52.608 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-11-08T12:26:52.609 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-11-08T12:26:52.748 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-11-08T12:26:52.843 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-fuse-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:26:53.248 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:26:53.528 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-fuse-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:26:53.528 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:53.529 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-11-08T12:26:53.529 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:26:53.529 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:53.529 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:26:53.796 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install rbd-mirror 2023-11-08T12:26:54.410 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:39 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:26:54.562 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:26:54.562 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:54.562 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repo Size 2023-11-08T12:26:54.562 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:54.563 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-11-08T12:26:54.563 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror x86_64 2:18.0.0-7227.g25e18238.el9 ceph 3.0 M 2023-11-08T12:26:54.563 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:54.563 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-11-08T12:26:54.563 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:54.564 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2023-11-08T12:26:54.564 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:54.564 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 3.0 M 2023-11-08T12:26:54.565 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 13 M 2023-11-08T12:26:54.565 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-11-08T12:26:54.922 INFO:teuthology.orchestra.run.smithi039.stdout:rbd-mirror-18.0.0-7227.g25e18238.el9.x86_64.rpm 8.3 MB/s | 3.0 MB 00:00 2023-11-08T12:26:54.923 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:26:54.924 INFO:teuthology.orchestra.run.smithi039.stdout:Total 8.3 MB/s | 3.0 MB 00:00 2023-11-08T12:26:54.924 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-11-08T12:26:54.933 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-11-08T12:26:54.933 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-11-08T12:26:55.009 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-11-08T12:26:55.009 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-11-08T12:26:55.460 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-11-08T12:26:55.514 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-mirror-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:26:55.541 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:26:55.541 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-08T12:26:55.542 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-11-08T12:26:55.542 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-08T12:26:55.543 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-08T12:26:55.543 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-mirror-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:26:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:56.730 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-11-08T12:26:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:26:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:56.731 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:26:56.990 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install rbd-nbd 2023-11-08T12:26:57.586 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:02:42 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:26:57.740 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:26:57.741 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:57.741 INFO:teuthology.orchestra.run.smithi039.stdout: Package Architecture Version Repository Size 2023-11-08T12:26:57.742 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:57.742 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-11-08T12:26:57.742 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd x86_64 2:18.0.0-7227.g25e18238.el9 ceph 169 k 2023-11-08T12:26:57.742 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:57.742 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-11-08T12:26:57.743 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:26:57.743 INFO:teuthology.orchestra.run.smithi039.stdout:Install 1 Package 2023-11-08T12:26:57.743 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:57.743 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 169 k 2023-11-08T12:26:57.743 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 537 k 2023-11-08T12:26:57.744 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-11-08T12:26:58.242 INFO:teuthology.orchestra.run.smithi039.stdout:rbd-nbd-18.0.0-7227.g25e18238.el9.x86_64.rpm 338 kB/s | 169 kB 00:00 2023-11-08T12:26:58.243 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:26:58.244 INFO:teuthology.orchestra.run.smithi039.stdout:Total 337 kB/s | 169 kB 00:00 2023-11-08T12:26:58.244 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-11-08T12:26:58.251 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-11-08T12:26:58.252 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-11-08T12:26:58.311 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-11-08T12:26:58.311 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-11-08T12:26:58.467 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-11-08T12:26:58.568 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-nbd-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:26:59.044 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:26:59.335 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-nbd-2:18.0.0-7227.g25e18238.el9.x86_64 1/1 2023-11-08T12:26:59.336 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:59.336 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-11-08T12:26:59.336 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:26:59.336 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:26:59.337 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:26:59.615 DEBUG:teuthology.parallel:result is None 2023-11-08T12:26:59.616 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=25e18238395a1cded51f624e1e84ec38cbcb1b4a 2023-11-08T12:26:59.755 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-08T12:26:59.777 INFO:teuthology.orchestra.run.smithi039.stdout:18.0.0-7227.g25e18238.el9 2023-11-08T12:26:59.778 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7227.g25e18238.el9 2023-11-08T12:26:59.778 INFO:teuthology.task.install:The correct ceph version 18.0.0-7227.g25e18238 is installed. 2023-11-08T12:26:59.779 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-08T12:26:59.779 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-11-08T12:26:59.780 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-08T12:26:59.849 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-08T12:26:59.850 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-11-08T12:26:59.850 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/daemon-helper 2023-11-08T12:26:59.922 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-08T12:26:59.993 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-08T12:26:59.994 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-11-08T12:26:59.994 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-08T12:27:00.064 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-08T12:27:00.135 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-08T12:27:00.136 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-11-08T12:27:00.136 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/stdin-killer 2023-11-08T12:27:00.206 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-08T12:27:00.278 INFO:teuthology.run_tasks:Running task ceph... 2023-11-08T12:27:00.347 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-08T12:27:00.348 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 777 /var/log/ceph 2023-11-08T12:27:00.378 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-08T12:27:00.379 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-08T12:27:00.449 INFO:tasks.ceph:Creating extra log directories... 2023-11-08T12:27:00.450 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-08T12:27:00.530 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-11-08T12:27:00.531 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-08T12:27:00.531 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2023-11-08_11:46:17-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7451942', 'branch': 'wip-matanb-crimson-do_osd_ops_execute-v3', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench}', 'email': None, 'first_in_suite': False, 'job_id': '7451942', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2023-11-08_11:46:17-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-do_osd_ops_execute-v3'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '25e18238395a1cded51f624e1e84ec38cbcb1b4a'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '25e18238395a1cded51f624e1e84ec38cbcb1b4a'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'db6fbc96537be451d7eb9f6302778fac84588cb6'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 6890, 'sha1': '25e18238395a1cded51f624e1e84ec38cbcb1b4a', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_db6fbc96537be451d7eb9f6302778fac84588cb6/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': 'db6fbc96537be451d7eb9f6302778fac84588cb6', 'targets': {'smithi039.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE4V3tEy6fAGYFySNX4MVgLuAvkFpJGmhe6LcFBY8bcYpEM9N8OiG+RWqCOAapFLw4rsALyn0JrQSrzlsTWQ1x8='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '25e18238395a1cded51f624e1e84ec38cbcb1b4a', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-11-08_11:46:17', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3590714'} 2023-11-08T12:27:00.532 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-08T12:27:00.579 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m0777 -- /var/run/ceph 2023-11-08T12:27:00.649 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-11-08T12:27:00.650 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/scratch_devs of=/dev/stdout 2023-11-08T12:27:00.705 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-08T12:27:00.706 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_1 2023-11-08T12:27:00.760 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-08T12:27:00.761 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-08T12:27:00.761 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 665 Links: 1 2023-11-08T12:27:00.761 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-08T12:27:00.761 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2023-11-08T12:27:00.762 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2023-11-08 12:26:55.802064048 +0000 2023-11-08T12:27:00.762 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2023-11-08 12:23:18.198999672 +0000 2023-11-08T12:27:00.762 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2023-11-08 12:23:18.198999672 +0000 2023-11-08T12:27:00.762 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2023-11-08 12:23:18.198999672 +0000 2023-11-08T12:27:00.763 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-08T12:27:00.830 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2023-11-08T12:27:00.830 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2023-11-08T12:27:00.831 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000291269 s, 1.8 MB/s 2023-11-08T12:27:00.832 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-08T12:27:00.888 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_2 2023-11-08T12:27:00.943 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-08T12:27:00.943 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-08T12:27:00.943 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 684 Links: 1 2023-11-08T12:27:00.943 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-08T12:27:00.944 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2023-11-08T12:27:00.944 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2023-11-08 12:26:55.802064048 +0000 2023-11-08T12:27:00.944 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2023-11-08 12:23:18.703989488 +0000 2023-11-08T12:27:00.944 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2023-11-08 12:23:18.703989488 +0000 2023-11-08T12:27:00.944 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2023-11-08 12:23:18.703989488 +0000 2023-11-08T12:27:00.945 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-08T12:27:01.011 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2023-11-08T12:27:01.012 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2023-11-08T12:27:01.012 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.00023137 s, 2.2 MB/s 2023-11-08T12:27:01.013 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-08T12:27:01.070 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_3 2023-11-08T12:27:01.125 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-08T12:27:01.125 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-08T12:27:01.125 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-11-08T12:27:01.126 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-08T12:27:01.126 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2023-11-08T12:27:01.126 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2023-11-08 12:26:55.803064028 +0000 2023-11-08T12:27:01.126 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2023-11-08 12:23:19.229978881 +0000 2023-11-08T12:27:01.127 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2023-11-08 12:23:19.229978881 +0000 2023-11-08T12:27:01.127 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2023-11-08 12:23:19.229978881 +0000 2023-11-08T12:27:01.127 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-08T12:27:01.195 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2023-11-08T12:27:01.195 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2023-11-08T12:27:01.195 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000328461 s, 1.6 MB/s 2023-11-08T12:27:01.196 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-08T12:27:01.252 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_4 2023-11-08T12:27:01.310 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-08T12:27:01.310 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-08T12:27:01.310 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 721 Links: 1 2023-11-08T12:27:01.310 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-08T12:27:01.311 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2023-11-08T12:27:01.311 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2023-11-08 12:26:55.803064028 +0000 2023-11-08T12:27:01.311 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2023-11-08 12:23:19.742968536 +0000 2023-11-08T12:27:01.311 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2023-11-08 12:23:19.742968536 +0000 2023-11-08T12:27:01.311 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2023-11-08 12:23:19.742968536 +0000 2023-11-08T12:27:01.312 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-08T12:27:01.379 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2023-11-08T12:27:01.379 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2023-11-08T12:27:01.379 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000273595 s, 1.9 MB/s 2023-11-08T12:27:01.380 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-08T12:27:01.436 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-08T12:27:01.437 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi039.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-08T12:27:01.437 INFO:tasks.ceph:Generating config... 2023-11-08T12:27:01.438 INFO:tasks.ceph:[osd] debug monc = 20 2023-11-08T12:27:01.438 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-08T12:27:01.439 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2023-11-08T12:27:01.439 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-11-08T12:27:01.439 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-08T12:27:01.440 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2023-11-08T12:27:01.440 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2023-11-08T12:27:01.440 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-11-08T12:27:01.440 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-11-08T12:27:01.441 INFO:tasks.ceph:[osd] debug filestore = 0/0 2023-11-08T12:27:01.441 INFO:tasks.ceph:[osd] debug journal = 0/0 2023-11-08T12:27:01.441 INFO:tasks.ceph:[osd] debug ms = 0/0 2023-11-08T12:27:01.441 INFO:tasks.ceph:[osd] debug osd = 0/0 2023-11-08T12:27:01.442 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-11-08T12:27:01.442 INFO:tasks.ceph:[osd] err to stderr = True 2023-11-08T12:27:01.442 INFO:tasks.ceph:[osd] log flush on exit = True 2023-11-08T12:27:01.442 INFO:tasks.ceph:[osd] log to file = False 2023-11-08T12:27:01.443 INFO:tasks.ceph:[osd] log to stderr = True 2023-11-08T12:27:01.443 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-08T12:27:01.443 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-08T12:27:01.443 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-08T12:27:01.444 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-08T12:27:01.444 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-08T12:27:01.444 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-11-08T12:27:01.444 INFO:tasks.ceph:[global] auth client required = none 2023-11-08T12:27:01.445 INFO:tasks.ceph:[global] auth cluster required = none 2023-11-08T12:27:01.445 INFO:tasks.ceph:[global] auth service required = none 2023-11-08T12:27:01.445 INFO:tasks.ceph:[global] auth supported = none 2023-11-08T12:27:01.445 INFO:tasks.ceph:[global] debug asok = 0/0 2023-11-08T12:27:01.446 INFO:tasks.ceph:[global] debug auth = 0/0 2023-11-08T12:27:01.446 INFO:tasks.ceph:[global] debug bdev = 0/0 2023-11-08T12:27:01.446 INFO:tasks.ceph:[global] debug bluefs = 0/0 2023-11-08T12:27:01.446 INFO:tasks.ceph:[global] debug bluestore = 0/0 2023-11-08T12:27:01.447 INFO:tasks.ceph:[global] debug buffer = 0/0 2023-11-08T12:27:01.447 INFO:tasks.ceph:[global] debug civetweb = 0/0 2023-11-08T12:27:01.447 INFO:tasks.ceph:[global] debug client = 0/0 2023-11-08T12:27:01.447 INFO:tasks.ceph:[global] debug compressor = 0/0 2023-11-08T12:27:01.448 INFO:tasks.ceph:[global] debug context = 0/0 2023-11-08T12:27:01.448 INFO:tasks.ceph:[global] debug crush = 0/0 2023-11-08T12:27:01.448 INFO:tasks.ceph:[global] debug crypto = 0/0 2023-11-08T12:27:01.448 INFO:tasks.ceph:[global] debug dpdk = 0/0 2023-11-08T12:27:01.449 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2023-11-08T12:27:01.449 INFO:tasks.ceph:[global] debug filer = 0/0 2023-11-08T12:27:01.449 INFO:tasks.ceph:[global] debug filestore = 0/0 2023-11-08T12:27:01.449 INFO:tasks.ceph:[global] debug finisher = 0/0 2023-11-08T12:27:01.450 INFO:tasks.ceph:[global] debug fuse = 0/0 2023-11-08T12:27:01.450 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2023-11-08T12:27:01.450 INFO:tasks.ceph:[global] debug javaclient = 0/0 2023-11-08T12:27:01.450 INFO:tasks.ceph:[global] debug journal = 0/0 2023-11-08T12:27:01.451 INFO:tasks.ceph:[global] debug journaler = 0/0 2023-11-08T12:27:01.451 INFO:tasks.ceph:[global] debug kstore = 0/0 2023-11-08T12:27:01.451 INFO:tasks.ceph:[global] debug leveldb = 0/0 2023-11-08T12:27:01.451 INFO:tasks.ceph:[global] debug lockdep = 0/0 2023-11-08T12:27:01.452 INFO:tasks.ceph:[global] debug mds = 0/0 2023-11-08T12:27:01.452 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2023-11-08T12:27:01.452 INFO:tasks.ceph:[global] debug mds locker = 0/0 2023-11-08T12:27:01.452 INFO:tasks.ceph:[global] debug mds log = 0/0 2023-11-08T12:27:01.453 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2023-11-08T12:27:01.453 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2023-11-08T12:27:01.453 INFO:tasks.ceph:[global] debug memdb = 0/0 2023-11-08T12:27:01.453 INFO:tasks.ceph:[global] debug mgr = 0/0 2023-11-08T12:27:01.454 INFO:tasks.ceph:[global] debug mgrc = 0/0 2023-11-08T12:27:01.454 INFO:tasks.ceph:[global] debug mon = 0/0 2023-11-08T12:27:01.454 INFO:tasks.ceph:[global] debug monc = 0/0 2023-11-08T12:27:01.454 INFO:tasks.ceph:[global] debug ms = 0/0 2023-11-08T12:27:01.455 INFO:tasks.ceph:[global] debug objclass = 0/0 2023-11-08T12:27:01.455 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2023-11-08T12:27:01.455 INFO:tasks.ceph:[global] debug objecter = 0/0 2023-11-08T12:27:01.455 INFO:tasks.ceph:[global] debug optracker = 0/0 2023-11-08T12:27:01.456 INFO:tasks.ceph:[global] debug osd = 0/0 2023-11-08T12:27:01.456 INFO:tasks.ceph:[global] debug paxos = 0/0 2023-11-08T12:27:01.456 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2023-11-08T12:27:01.456 INFO:tasks.ceph:[global] debug rados = 0/0 2023-11-08T12:27:01.457 INFO:tasks.ceph:[global] debug rbd = 0/0 2023-11-08T12:27:01.457 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2023-11-08T12:27:01.457 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2023-11-08T12:27:01.457 INFO:tasks.ceph:[global] debug refs = 0/0 2023-11-08T12:27:01.457 INFO:tasks.ceph:[global] debug rgw = 0/0 2023-11-08T12:27:01.458 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2023-11-08T12:27:01.458 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2023-11-08T12:27:01.458 INFO:tasks.ceph:[global] debug striper = 0/0 2023-11-08T12:27:01.458 INFO:tasks.ceph:[global] debug throttle = 0/0 2023-11-08T12:27:01.459 INFO:tasks.ceph:[global] debug timer = 0/0 2023-11-08T12:27:01.459 INFO:tasks.ceph:[global] debug tp = 0/0 2023-11-08T12:27:01.459 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2023-11-08T12:27:01.460 INFO:tasks.ceph:[global] ms client mode = crc 2023-11-08T12:27:01.460 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-11-08T12:27:01.460 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-11-08T12:27:01.460 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-11-08T12:27:01.461 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-11-08T12:27:01.461 INFO:tasks.ceph:[global] ms service mode = crc 2023-11-08T12:27:01.461 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-08T12:27:01.461 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-08T12:27:01.462 INFO:tasks.ceph:[mon] debug mon = 0/0 2023-11-08T12:27:01.462 INFO:tasks.ceph:[mon] debug ms = 0/0 2023-11-08T12:27:01.462 INFO:tasks.ceph:[mon] debug paxos = 0/0 2023-11-08T12:27:01.462 INFO:tasks.ceph:[mon] osd pool default crimson = True 2023-11-08T12:27:01.463 INFO:tasks.ceph:Setting up mon.a... 2023-11-08T12:27:01.463 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-11-08T12:27:01.552 INFO:teuthology.orchestra.run.smithi039.stdout:creating /etc/ceph/ceph.keyring 2023-11-08T12:27:01.555 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-11-08T12:27:01.608 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-08T12:27:01.639 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.39')] 2023-11-08T12:27:01.640 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.39', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-11-08T12:27:01.641 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-11-08T12:27:01.641 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-08T12:27:01.694 DEBUG:teuthology.orchestra.run.smithi039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.39 --print /home/ubuntu/cephtest/ceph.monmap 2023-11-08T12:27:01.781 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:01.779+0000 7f62e702ce80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:01.779+0000 7f62e702ce80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:01.779+0000 7f62e702ce80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:01.783 INFO:teuthology.orchestra.run.smithi039.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-11-08T12:27:01.783 INFO:teuthology.orchestra.run.smithi039.stdout:monmaptool: generated fsid a1557d23-2394-44bb-8ae3-ddbb181f1a30 2023-11-08T12:27:01.784 INFO:teuthology.orchestra.run.smithi039.stdout:setting min_mon_release = pacific 2023-11-08T12:27:01.784 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 0 2023-11-08T12:27:01.784 INFO:teuthology.orchestra.run.smithi039.stdout:fsid a1557d23-2394-44bb-8ae3-ddbb181f1a30 2023-11-08T12:27:01.784 INFO:teuthology.orchestra.run.smithi039.stdout:last_changed 2023-11-08T12:27:01.781431+0000 2023-11-08T12:27:01.784 INFO:teuthology.orchestra.run.smithi039.stdout:created 2023-11-08T12:27:01.781431+0000 2023-11-08T12:27:01.785 INFO:teuthology.orchestra.run.smithi039.stdout:min_mon_release 16 (pacific) 2023-11-08T12:27:01.785 INFO:teuthology.orchestra.run.smithi039.stdout:election_strategy: 1 2023-11-08T12:27:01.785 INFO:teuthology.orchestra.run.smithi039.stdout:0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2023-11-08T12:27:01.785 INFO:teuthology.orchestra.run.smithi039.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-11-08T12:27:01.786 DEBUG:teuthology.orchestra.run.smithi039:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-08T12:27:01.802 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a1557d23-2394-44bb-8ae3-ddbb181f1a30... 2023-11-08T12:27:01.803 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-08T12:27:01.903 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2023-11-08T12:27:01.903 INFO:teuthology.orchestra.run.smithi039.stdout: chdir = "" 2023-11-08T12:27:01.903 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-08T12:27:01.904 INFO:teuthology.orchestra.run.smithi039.stdout: auth supported = none 2023-11-08T12:27:01.904 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.904 INFO:teuthology.orchestra.run.smithi039.stdout: filestore xattr use omap = true 2023-11-08T12:27:01.904 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.904 INFO:teuthology.orchestra.run.smithi039.stdout: mon clock drift allowed = 1.000 2023-11-08T12:27:01.905 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.905 INFO:teuthology.orchestra.run.smithi039.stdout: osd crush chooseleaf type = 0 2023-11-08T12:27:01.905 INFO:teuthology.orchestra.run.smithi039.stdout: auth debug = true 2023-11-08T12:27:01.905 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.906 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on old message = true 2023-11-08T12:27:01.906 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on bug = true 2023-11-08T12:27:01.906 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.906 INFO:teuthology.orchestra.run.smithi039.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-08T12:27:01.907 INFO:teuthology.orchestra.run.smithi039.stdout: mon pg warn max object skew = 0 2023-11-08T12:27:01.907 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.907 INFO:teuthology.orchestra.run.smithi039.stdout: # disable pg_autoscaler by default for new pools 2023-11-08T12:27:01.907 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-08T12:27:01.907 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.908 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default size = 2 2023-11-08T12:27:01.908 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.908 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow primary affinity = true 2023-11-08T12:27:01.908 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow pg remap = true 2023-11-08T12:27:01.909 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on legacy crush tunables = false 2023-11-08T12:27:01.909 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on crush straw calc version zero = false 2023-11-08T12:27:01.909 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on no sortbitwise = false 2023-11-08T12:27:01.909 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on osd down out interval zero = false 2023-11-08T12:27:01.909 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on too few osds = false 2023-11-08T12:27:01.910 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-08T12:27:01.910 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-08T12:27:01.910 INFO:teuthology.orchestra.run.smithi039.stdout: mon_allow_pool_size_one = true 2023-11-08T12:27:01.910 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.910 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-08T12:27:01.911 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.911 INFO:teuthology.orchestra.run.smithi039.stdout: osd default data pool replay window = 5 2023-11-08T12:27:01.911 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.911 INFO:teuthology.orchestra.run.smithi039.stdout: mon allow pool delete = true 2023-11-08T12:27:01.911 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.912 INFO:teuthology.orchestra.run.smithi039.stdout: mon cluster log file level = debug 2023-11-08T12:27:01.912 INFO:teuthology.orchestra.run.smithi039.stdout: debug asserts on shutdown = true 2023-11-08T12:27:01.912 INFO:teuthology.orchestra.run.smithi039.stdout: mon health detail to clog = false 2023-11-08T12:27:01.912 INFO:teuthology.orchestra.run.smithi039.stdout: mon host = 172.21.15.39 2023-11-08T12:27:01.912 INFO:teuthology.orchestra.run.smithi039.stdout: auth client required = none 2023-11-08T12:27:01.913 INFO:teuthology.orchestra.run.smithi039.stdout: auth cluster required = none 2023-11-08T12:27:01.913 INFO:teuthology.orchestra.run.smithi039.stdout: auth service required = none 2023-11-08T12:27:01.913 INFO:teuthology.orchestra.run.smithi039.stdout: debug asok = 0/0 2023-11-08T12:27:01.913 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 0/0 2023-11-08T12:27:01.913 INFO:teuthology.orchestra.run.smithi039.stdout: debug bdev = 0/0 2023-11-08T12:27:01.913 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluefs = 0/0 2023-11-08T12:27:01.914 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluestore = 0/0 2023-11-08T12:27:01.914 INFO:teuthology.orchestra.run.smithi039.stdout: debug buffer = 0/0 2023-11-08T12:27:01.914 INFO:teuthology.orchestra.run.smithi039.stdout: debug civetweb = 0/0 2023-11-08T12:27:01.914 INFO:teuthology.orchestra.run.smithi039.stdout: debug client = 0/0 2023-11-08T12:27:01.914 INFO:teuthology.orchestra.run.smithi039.stdout: debug compressor = 0/0 2023-11-08T12:27:01.915 INFO:teuthology.orchestra.run.smithi039.stdout: debug context = 0/0 2023-11-08T12:27:01.915 INFO:teuthology.orchestra.run.smithi039.stdout: debug crush = 0/0 2023-11-08T12:27:01.915 INFO:teuthology.orchestra.run.smithi039.stdout: debug crypto = 0/0 2023-11-08T12:27:01.915 INFO:teuthology.orchestra.run.smithi039.stdout: debug dpdk = 0/0 2023-11-08T12:27:01.915 INFO:teuthology.orchestra.run.smithi039.stdout: debug eventtrace = 0/0 2023-11-08T12:27:01.916 INFO:teuthology.orchestra.run.smithi039.stdout: debug filer = 0/0 2023-11-08T12:27:01.916 INFO:teuthology.orchestra.run.smithi039.stdout: debug filestore = 0/0 2023-11-08T12:27:01.916 INFO:teuthology.orchestra.run.smithi039.stdout: debug finisher = 0/0 2023-11-08T12:27:01.917 INFO:teuthology.orchestra.run.smithi039.stdout: debug fuse = 0/0 2023-11-08T12:27:01.917 INFO:teuthology.orchestra.run.smithi039.stdout: debug heartbeatmap = 0/0 2023-11-08T12:27:01.917 INFO:teuthology.orchestra.run.smithi039.stdout: debug javaclient = 0/0 2023-11-08T12:27:01.917 INFO:teuthology.orchestra.run.smithi039.stdout: debug journal = 0/0 2023-11-08T12:27:01.917 INFO:teuthology.orchestra.run.smithi039.stdout: debug journaler = 0/0 2023-11-08T12:27:01.918 INFO:teuthology.orchestra.run.smithi039.stdout: debug kstore = 0/0 2023-11-08T12:27:01.918 INFO:teuthology.orchestra.run.smithi039.stdout: debug leveldb = 0/0 2023-11-08T12:27:01.918 INFO:teuthology.orchestra.run.smithi039.stdout: debug lockdep = 0/0 2023-11-08T12:27:01.918 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds = 0/0 2023-11-08T12:27:01.918 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds balancer = 0/0 2023-11-08T12:27:01.918 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds locker = 0/0 2023-11-08T12:27:01.919 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds log = 0/0 2023-11-08T12:27:01.919 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds log expire = 0/0 2023-11-08T12:27:01.919 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds migrator = 0/0 2023-11-08T12:27:01.919 INFO:teuthology.orchestra.run.smithi039.stdout: debug memdb = 0/0 2023-11-08T12:27:01.920 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgr = 0/0 2023-11-08T12:27:01.920 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgrc = 0/0 2023-11-08T12:27:01.920 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 0/0 2023-11-08T12:27:01.920 INFO:teuthology.orchestra.run.smithi039.stdout: debug monc = 0/0 2023-11-08T12:27:01.920 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 0/0 2023-11-08T12:27:01.921 INFO:teuthology.orchestra.run.smithi039.stdout: debug objclass = 0/0 2023-11-08T12:27:01.921 INFO:teuthology.orchestra.run.smithi039.stdout: debug objectcacher = 0/0 2023-11-08T12:27:01.921 INFO:teuthology.orchestra.run.smithi039.stdout: debug objecter = 0/0 2023-11-08T12:27:01.921 INFO:teuthology.orchestra.run.smithi039.stdout: debug optracker = 0/0 2023-11-08T12:27:01.921 INFO:teuthology.orchestra.run.smithi039.stdout: debug osd = 0/0 2023-11-08T12:27:01.922 INFO:teuthology.orchestra.run.smithi039.stdout: debug paxos = 0/0 2023-11-08T12:27:01.922 INFO:teuthology.orchestra.run.smithi039.stdout: debug perfcounter = 0/0 2023-11-08T12:27:01.922 INFO:teuthology.orchestra.run.smithi039.stdout: debug rados = 0/0 2023-11-08T12:27:01.922 INFO:teuthology.orchestra.run.smithi039.stdout: debug rbd = 0/0 2023-11-08T12:27:01.922 INFO:teuthology.orchestra.run.smithi039.stdout: debug rbd mirror = 0/0 2023-11-08T12:27:01.923 INFO:teuthology.orchestra.run.smithi039.stdout: debug rbd replay = 0/0 2023-11-08T12:27:01.923 INFO:teuthology.orchestra.run.smithi039.stdout: debug refs = 0/0 2023-11-08T12:27:01.923 INFO:teuthology.orchestra.run.smithi039.stdout: debug rgw = 0/0 2023-11-08T12:27:01.923 INFO:teuthology.orchestra.run.smithi039.stdout: debug rgw sync = 0/0 2023-11-08T12:27:01.923 INFO:teuthology.orchestra.run.smithi039.stdout: debug rocksdb = 0/0 2023-11-08T12:27:01.924 INFO:teuthology.orchestra.run.smithi039.stdout: debug striper = 0/0 2023-11-08T12:27:01.924 INFO:teuthology.orchestra.run.smithi039.stdout: debug throttle = 0/0 2023-11-08T12:27:01.924 INFO:teuthology.orchestra.run.smithi039.stdout: debug timer = 0/0 2023-11-08T12:27:01.924 INFO:teuthology.orchestra.run.smithi039.stdout: debug tp = 0/0 2023-11-08T12:27:01.924 INFO:teuthology.orchestra.run.smithi039.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-11-08T12:27:01.924 INFO:teuthology.orchestra.run.smithi039.stdout: ms client mode = crc 2023-11-08T12:27:01.925 INFO:teuthology.orchestra.run.smithi039.stdout: ms cluster mode = crc 2023-11-08T12:27:01.925 INFO:teuthology.orchestra.run.smithi039.stdout: ms mon client mode = crc 2023-11-08T12:27:01.925 INFO:teuthology.orchestra.run.smithi039.stdout: ms mon cluster mode = crc 2023-11-08T12:27:01.925 INFO:teuthology.orchestra.run.smithi039.stdout: ms mon service mode = crc 2023-11-08T12:27:01.925 INFO:teuthology.orchestra.run.smithi039.stdout: ms service mode = crc 2023-11-08T12:27:01.926 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = a1557d23-2394-44bb-8ae3-ddbb181f1a30 2023-11-08T12:27:01.926 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.926 INFO:teuthology.orchestra.run.smithi039.stdout:[osd] 2023-11-08T12:27:01.926 INFO:teuthology.orchestra.run.smithi039.stdout: osd journal size = 100 2023-11-08T12:27:01.927 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.927 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub load threshold = 5.0 2023-11-08T12:27:01.927 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub max interval = 600 2023-11-08T12:27:01.927 INFO:teuthology.orchestra.run.smithi039.stdout: osd mclock profile = high_recovery_ops 2023-11-08T12:27:01.927 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.928 INFO:teuthology.orchestra.run.smithi039.stdout: osd recover clone overlap = true 2023-11-08T12:27:01.928 INFO:teuthology.orchestra.run.smithi039.stdout: osd recovery max chunk = 1048576 2023-11-08T12:27:01.928 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.928 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug shutdown = true 2023-11-08T12:27:01.928 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug op order = true 2023-11-08T12:27:01.928 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify stray on activate = true 2023-11-08T12:27:01.929 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug trim objects = true 2023-11-08T12:27:01.929 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.929 INFO:teuthology.orchestra.run.smithi039.stdout: osd open classes on start = true 2023-11-08T12:27:01.929 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug pg log writeout = true 2023-11-08T12:27:01.929 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.930 INFO:teuthology.orchestra.run.smithi039.stdout: osd deep scrub update digest min age = 30 2023-11-08T12:27:01.930 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.930 INFO:teuthology.orchestra.run.smithi039.stdout: osd map max advance = 10 2023-11-08T12:27:01.930 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.930 INFO:teuthology.orchestra.run.smithi039.stdout: journal zero on create = true 2023-11-08T12:27:01.931 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.931 INFO:teuthology.orchestra.run.smithi039.stdout: filestore ondisk finisher threads = 3 2023-11-08T12:27:01.931 INFO:teuthology.orchestra.run.smithi039.stdout: filestore apply finisher threads = 3 2023-11-08T12:27:01.931 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.931 INFO:teuthology.orchestra.run.smithi039.stdout: bdev debug aio = true 2023-11-08T12:27:01.932 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug misdirected ops = true 2023-11-08T12:27:01.932 INFO:teuthology.orchestra.run.smithi039.stdout: debug monc = 20 2023-11-08T12:27:01.932 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore block size = 96636764160 2023-11-08T12:27:01.932 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore compression algorithm = snappy 2023-11-08T12:27:01.933 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore compression mode = aggressive 2023-11-08T12:27:01.933 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore fsck on mount = True 2023-11-08T12:27:01.933 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore rocksdb cf = False 2023-11-08T12:27:01.933 INFO:teuthology.orchestra.run.smithi039.stdout: crimson seastar smp = 3 2023-11-08T12:27:01.934 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluefs = 20 2023-11-08T12:27:01.934 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluestore = 20 2023-11-08T12:27:01.934 INFO:teuthology.orchestra.run.smithi039.stdout: debug filestore = 0/0 2023-11-08T12:27:01.934 INFO:teuthology.orchestra.run.smithi039.stdout: debug journal = 0/0 2023-11-08T12:27:01.934 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 0/0 2023-11-08T12:27:01.935 INFO:teuthology.orchestra.run.smithi039.stdout: debug osd = 0/0 2023-11-08T12:27:01.935 INFO:teuthology.orchestra.run.smithi039.stdout: debug rocksdb = 10 2023-11-08T12:27:01.935 INFO:teuthology.orchestra.run.smithi039.stdout: err to stderr = True 2023-11-08T12:27:01.935 INFO:teuthology.orchestra.run.smithi039.stdout: log flush on exit = True 2023-11-08T12:27:01.935 INFO:teuthology.orchestra.run.smithi039.stdout: log to file = False 2023-11-08T12:27:01.936 INFO:teuthology.orchestra.run.smithi039.stdout: log to stderr = True 2023-11-08T12:27:01.936 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd backfillfull_ratio = 0.85 2023-11-08T12:27:01.936 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd full ratio = 0.9 2023-11-08T12:27:01.936 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd nearfull ratio = 0.8 2023-11-08T12:27:01.936 INFO:teuthology.orchestra.run.smithi039.stdout: osd failsafe full ratio = 0.95 2023-11-08T12:27:01.937 INFO:teuthology.orchestra.run.smithi039.stdout: osd objectstore = bluestore 2023-11-08T12:27:01.937 INFO:teuthology.orchestra.run.smithi039.stdout: osd shutdown pgref assert = True 2023-11-08T12:27:01.937 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.937 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr] 2023-11-08T12:27:01.938 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2023-11-08T12:27:01.938 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgr = 20 2023-11-08T12:27:01.938 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2023-11-08T12:27:01.938 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2023-11-08T12:27:01.938 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2023-11-08T12:27:01.939 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2023-11-08T12:27:01.939 INFO:teuthology.orchestra.run.smithi039.stdout: mgr/telemetry/nag = false 2023-11-08T12:27:01.939 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.939 INFO:teuthology.orchestra.run.smithi039.stdout:[mon] 2023-11-08T12:27:01.939 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 0/0 2023-11-08T12:27:01.940 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 0/0 2023-11-08T12:27:01.940 INFO:teuthology.orchestra.run.smithi039.stdout: debug paxos = 0/0 2023-11-08T12:27:01.940 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2023-11-08T12:27:01.940 INFO:teuthology.orchestra.run.smithi039.stdout: mon data avail warn = 5 2023-11-08T12:27:01.940 INFO:teuthology.orchestra.run.smithi039.stdout: mon mgr mkfs grace = 240 2023-11-08T12:27:01.941 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2023-11-08T12:27:01.941 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd reporter subtree level = osd 2023-11-08T12:27:01.941 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd prime pg temp = true 2023-11-08T12:27:01.941 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2023-11-08T12:27:01.941 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.942 INFO:teuthology.orchestra.run.smithi039.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-08T12:27:01.942 INFO:teuthology.orchestra.run.smithi039.stdout: auth mon ticket ttl = 660 # 11m 2023-11-08T12:27:01.942 INFO:teuthology.orchestra.run.smithi039.stdout: auth service ticket ttl = 240 # 4m 2023-11-08T12:27:01.942 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.943 INFO:teuthology.orchestra.run.smithi039.stdout: # don't complain about insecure global_id in the test suite 2023-11-08T12:27:01.943 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-08T12:27:01.943 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-08T12:27:01.943 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.943 INFO:teuthology.orchestra.run.smithi039.stdout: # 1m isn't quite enough 2023-11-08T12:27:01.944 INFO:teuthology.orchestra.run.smithi039.stdout: mon_down_mkfs_grace = 2m 2023-11-08T12:27:01.944 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.944 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_filestore_osds = false 2023-11-08T12:27:01.944 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default crimson = True 2023-11-08T12:27:01.944 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:01.944 INFO:teuthology.orchestra.run.smithi039.stdout:[client] 2023-11-08T12:27:01.945 INFO:teuthology.orchestra.run.smithi039.stdout: rgw cache enabled = true 2023-11-08T12:27:01.945 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable ops log = true 2023-11-08T12:27:01.945 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable usage log = true 2023-11-08T12:27:01.945 INFO:teuthology.orchestra.run.smithi039.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-08T12:27:01.945 INFO:teuthology.orchestra.run.smithi039.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-08T12:27:01.946 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.a] 2023-11-08T12:27:01.947 INFO:tasks.ceph:Creating admin key on mon.a... 2023-11-08T12:27:01.947 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-11-08T12:27:02.054 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-08T12:27:02.054 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-11-08T12:27:02.055 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-11-08T12:27:02.071 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-11-08T12:27:02.072 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-11-08T12:27:02.127 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:27:02.127 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-11-08T12:27:02.127 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-08T12:27:02.128 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-08T12:27:02.214 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-11-08T12:27:02.214 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-08T12:27:02.269 INFO:tasks.ceph:Setting up mon nodes... 2023-11-08T12:27:02.269 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-08T12:27:02.270 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-11-08T12:27:02.377 INFO:teuthology.orchestra.run.smithi039.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-11-08T12:27:02.380 INFO:tasks.ceph:Setting up mds nodes... 2023-11-08T12:27:02.381 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-08T12:27:02.382 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-08T12:27:02.434 INFO:teuthology.orchestra.run.smithi039.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-11-08T12:27:02.453 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-08T12:27:02.454 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi039.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-08T12:27:02.455 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-11-08T12:27:02.486 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-08T12:27:02.486 INFO:tasks.ceph:role: osd.0 2023-11-08T12:27:02.487 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:27:02.487 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-08T12:27:02.585 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-08T12:27:02.585 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-08T12:27:02.585 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-08T12:27:02.586 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-08T12:27:02.586 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-08T12:27:02.586 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2023-11-08T12:27:02.586 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-08T12:27:02.587 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-08T12:27:02.587 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-08T12:27:02.587 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-08T12:27:03.901 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2023-11-08T12:27:03.903 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2023-11-08T12:27:03.904 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-11-08T12:27:03.946 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-11-08T12:27:04.023 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-11-08T12:27:04.097 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-08T12:27:04.097 INFO:tasks.ceph:role: osd.1 2023-11-08T12:27:04.098 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:27:04.098 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-08T12:27:04.197 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-08T12:27:04.197 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-08T12:27:04.198 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-08T12:27:04.198 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-08T12:27:04.198 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-08T12:27:04.199 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2023-11-08T12:27:04.199 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-08T12:27:04.199 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-08T12:27:04.199 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-08T12:27:04.199 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-08T12:27:05.506 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2023-11-08T12:27:05.508 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2023-11-08T12:27:05.509 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-11-08T12:27:05.549 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-11-08T12:27:05.626 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-11-08T12:27:05.698 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-08T12:27:05.699 INFO:tasks.ceph:role: osd.2 2023-11-08T12:27:05.699 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:27:05.699 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-08T12:27:05.797 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-08T12:27:05.798 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-08T12:27:05.798 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-08T12:27:05.799 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-08T12:27:05.799 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-08T12:27:05.799 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2023-11-08T12:27:05.799 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-08T12:27:05.799 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-08T12:27:05.800 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-08T12:27:05.800 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-08T12:27:07.134 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2023-11-08T12:27:07.137 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2023-11-08T12:27:07.137 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-11-08T12:27:07.179 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-11-08T12:27:07.259 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-08T12:27:07.829 INFO:teuthology.orchestra.run.smithi039.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-08T12:27:07.829 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:07,829 seastar - Reactor backend: linux-aio 2023-11-08T12:27:07.845 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:07,845 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-08T12:27:07.845 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:07,845 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-08T12:27:07.846 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:07,845 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-08T12:27:07.846 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:07,845 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-08T12:27:07.847 INFO:teuthology.orchestra.run.smithi039.stderr:==79858==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-08T12:27:07.854 INFO:teuthology.orchestra.run.smithi039.stderr:==79858==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffcfbca1000; bottom 0x7f655886b000; size: 0x0097a3436000 (651279163392) 2023-11-08T12:27:07.854 INFO:teuthology.orchestra.run.smithi039.stderr:False positive error reports may follow 2023-11-08T12:27:07.854 INFO:teuthology.orchestra.run.smithi039.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-08T12:27:07.880 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0x22c560de 0x22cffadf 0x22cdd989 0x22cde040 0x22cf33ea 0x22cf5d39 0x22cf7ce8 0x22d1ca2c 0x22d3bca1 0x22d3c529 0x22cea0f9 0x22cf9719 0x22cfae15 0x22cfb7ee 0x22cfc8d9 0x22dde828 0x22c3eba9 0x22d0d764 0x22d0f1af 0x22d0ff18 0x22cd7d84 0x22cd73dc 0x229e96c4 0x24a1bd30 0x24a234e6 0x24a25e34 0x24a2700d 0x2258a2b7 0x225daf14 0x225fec49 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1ed04ea4 0x1ed0dc41 0x1eb80e12 0x3feaf 0x3ff5f 0x1e62ba44 2023-11-08T12:27:07.880 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2023-11-08T12:27:08.414 INFO:teuthology.orchestra.run.smithi039.stderr:----------------------------------------------------- 2023-11-08T12:27:08.415 INFO:teuthology.orchestra.run.smithi039.stderr:Suppressions used: 2023-11-08T12:27:08.415 INFO:teuthology.orchestra.run.smithi039.stderr: count bytes template 2023-11-08T12:27:08.415 INFO:teuthology.orchestra.run.smithi039.stderr: 1 8 MallocExtension::Register 2023-11-08T12:27:08.415 INFO:teuthology.orchestra.run.smithi039.stderr:----------------------------------------------------- 2023-11-08T12:27:08.415 INFO:teuthology.orchestra.run.smithi039.stderr: 2023-11-08T12:27:08.470 INFO:teuthology.orchestra.run.smithi039.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-08T12:27:08.471 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:08,471 seastar - Reactor backend: linux-aio 2023-11-08T12:27:08.489 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:08,489 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-08T12:27:08.490 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:08,489 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-08T12:27:08.490 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:08,489 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-08T12:27:08.497 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:08,496 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-08T12:27:08.499 INFO:teuthology.orchestra.run.smithi039.stderr:==79856==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-08T12:27:08.505 INFO:teuthology.orchestra.run.smithi039.stderr:==79856==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffcfbca1000; bottom 0x7f6556664000; size: 0x0097a563d000 (651314843648) 2023-11-08T12:27:08.505 INFO:teuthology.orchestra.run.smithi039.stderr:False positive error reports may follow 2023-11-08T12:27:08.506 INFO:teuthology.orchestra.run.smithi039.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-08T12:27:08.531 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0x1e7d9c2d 0x22b39ce2 0x22b3ea3f 0x22b440b0 0x24a1c7fb 0x24a234e6 0x24a25e34 0x24a2700d 0x2258a2b7 0x225daf14 0x225fec49 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1eb826c8 0x3feaf 0x3ff5f 0x1e62ba44 2023-11-08T12:27:08.532 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2023-11-08T12:27:08.594 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x26251dd8 0x26254421 0x26256232 0x26258a97 0x2608d333 0x1ec3ac08 0x1ebb0927 0x1ebb4768 0x2c9f947b 2023-11-08T12:27:08.594 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2023-11-08T12:27:12.556 INFO:teuthology.orchestra.run.smithi039.stderr:----------------------------------------------------- 2023-11-08T12:27:12.557 INFO:teuthology.orchestra.run.smithi039.stderr:Suppressions used: 2023-11-08T12:27:12.557 INFO:teuthology.orchestra.run.smithi039.stderr: count bytes template 2023-11-08T12:27:12.557 INFO:teuthology.orchestra.run.smithi039.stderr: 1 8 MallocExtension::Register 2023-11-08T12:27:12.557 INFO:teuthology.orchestra.run.smithi039.stderr:----------------------------------------------------- 2023-11-08T12:27:12.557 INFO:teuthology.orchestra.run.smithi039.stderr: 2023-11-08T12:27:12.558 INFO:teuthology.orchestra.run.smithi039.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid a1557d23-2394-44bb-8ae3-ddbb181f1a30 2023-11-08T12:27:12.595 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-11-08T12:27:12.628 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-08T12:27:13.177 INFO:teuthology.orchestra.run.smithi039.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-08T12:27:13.178 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:13,178 seastar - Reactor backend: linux-aio 2023-11-08T12:27:13.196 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:13,196 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-08T12:27:13.197 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:13,196 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-08T12:27:13.197 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:13,196 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-08T12:27:13.197 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:13,196 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-08T12:27:13.198 INFO:teuthology.orchestra.run.smithi039.stderr:==80040==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-08T12:27:13.206 INFO:teuthology.orchestra.run.smithi039.stderr:==80040==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe85344000; bottom 0x7f7da7516000; size: 0x0080dde2e000 (553478447104) 2023-11-08T12:27:13.207 INFO:teuthology.orchestra.run.smithi039.stderr:False positive error reports may follow 2023-11-08T12:27:13.207 INFO:teuthology.orchestra.run.smithi039.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-08T12:27:13.231 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0xba13f 0x2c6e3 0x2857c 0xb6323 0x22c08eb4 0x22c58ec2 0x22d33f6b 0x22d377e3 0x22d37b15 0x22d391cf 0x22d3c036 0x22d3c529 0x22cea0f9 0x22cf9719 0x22cfae15 0x22cfb7ee 0x22cfc8d9 0x22dde828 0x22c3eba9 0x22d0d764 0x22d0f1af 0x22d0ff18 0x22cd7d84 0x22cd73dc 0x229e96c4 0x24a1bd30 0x24a234e6 0x24a25e34 0x24a2700d 0x2258a2b7 0x225daf14 0x225fec49 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1ed04ea4 0x1ed0dc41 0x1eb80e12 0x3feaf 0x3ff5f 0x1e62ba44 2023-11-08T12:27:13.231 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2023-11-08T12:27:13.741 INFO:teuthology.orchestra.run.smithi039.stderr:----------------------------------------------------- 2023-11-08T12:27:13.741 INFO:teuthology.orchestra.run.smithi039.stderr:Suppressions used: 2023-11-08T12:27:13.741 INFO:teuthology.orchestra.run.smithi039.stderr: count bytes template 2023-11-08T12:27:13.742 INFO:teuthology.orchestra.run.smithi039.stderr: 1 8 MallocExtension::Register 2023-11-08T12:27:13.742 INFO:teuthology.orchestra.run.smithi039.stderr:----------------------------------------------------- 2023-11-08T12:27:13.742 INFO:teuthology.orchestra.run.smithi039.stderr: 2023-11-08T12:27:13.795 INFO:teuthology.orchestra.run.smithi039.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-08T12:27:13.796 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:13,795 seastar - Reactor backend: linux-aio 2023-11-08T12:27:13.814 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:13,813 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-08T12:27:13.814 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:13,813 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-08T12:27:13.814 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:13,813 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-08T12:27:13.821 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:13,821 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-08T12:27:13.824 INFO:teuthology.orchestra.run.smithi039.stderr:==80038==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-08T12:27:13.830 INFO:teuthology.orchestra.run.smithi039.stderr:==80038==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe85344000; bottom 0x7f7da530f000; size: 0x0080e0035000 (553514127360) 2023-11-08T12:27:13.830 INFO:teuthology.orchestra.run.smithi039.stderr:False positive error reports may follow 2023-11-08T12:27:13.830 INFO:teuthology.orchestra.run.smithi039.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-08T12:27:13.927 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x26251dd8 0x26254421 0x26256232 0x26258a97 0x2608d333 0x1ec3ac08 0x1ebb0927 0x1ebb4768 0x2c9f947b 2023-11-08T12:27:13.928 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2023-11-08T12:27:17.916 INFO:teuthology.orchestra.run.smithi039.stderr:----------------------------------------------------- 2023-11-08T12:27:17.917 INFO:teuthology.orchestra.run.smithi039.stderr:Suppressions used: 2023-11-08T12:27:17.917 INFO:teuthology.orchestra.run.smithi039.stderr: count bytes template 2023-11-08T12:27:17.917 INFO:teuthology.orchestra.run.smithi039.stderr: 1 8 MallocExtension::Register 2023-11-08T12:27:17.917 INFO:teuthology.orchestra.run.smithi039.stderr:----------------------------------------------------- 2023-11-08T12:27:17.917 INFO:teuthology.orchestra.run.smithi039.stderr: 2023-11-08T12:27:17.918 INFO:teuthology.orchestra.run.smithi039.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid a1557d23-2394-44bb-8ae3-ddbb181f1a30 2023-11-08T12:27:17.956 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-11-08T12:27:17.988 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-08T12:27:18.548 INFO:teuthology.orchestra.run.smithi039.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-08T12:27:18.549 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:18,548 seastar - Reactor backend: linux-aio 2023-11-08T12:27:18.567 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:18,567 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-08T12:27:18.568 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:18,567 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-08T12:27:18.568 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:18,567 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-08T12:27:18.568 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:18,567 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-08T12:27:18.569 INFO:teuthology.orchestra.run.smithi039.stderr:==80222==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-08T12:27:18.576 INFO:teuthology.orchestra.run.smithi039.stderr:==80222==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc8d47e000; bottom 0x7f4bb5d82000; size: 0x00b0d76fc000 (759528669184) 2023-11-08T12:27:18.576 INFO:teuthology.orchestra.run.smithi039.stderr:False positive error reports may follow 2023-11-08T12:27:18.576 INFO:teuthology.orchestra.run.smithi039.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-08T12:27:18.602 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0x2c605 0x2857c 0xb6323 0x22c08eb4 0x22c58ec2 0x22d33f6b 0x22d377e3 0x22d37b15 0x22d391cf 0x22d3c036 0x22d3c529 0x22cea0f9 0x22cf9719 0x22cfae15 0x22cfb7ee 0x22cfc8d9 0x22dde828 0x22c3eba9 0x22d0d764 0x22d0f1af 0x22d0ff18 0x22cd7d84 0x22cd73dc 0x229e96c4 0x24a1bd30 0x24a234e6 0x24a25e34 0x24a2700d 0x2258a2b7 0x225daf14 0x225fec49 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1ed04ea4 0x1ed0dc41 0x1eb80e12 0x3feaf 0x3ff5f 0x1e62ba44 2023-11-08T12:27:18.602 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff93089a14 0xffffffff93c65842 0xffffffff93e00b82 2023-11-08T12:27:19.121 INFO:teuthology.orchestra.run.smithi039.stderr:----------------------------------------------------- 2023-11-08T12:27:19.122 INFO:teuthology.orchestra.run.smithi039.stderr:Suppressions used: 2023-11-08T12:27:19.122 INFO:teuthology.orchestra.run.smithi039.stderr: count bytes template 2023-11-08T12:27:19.122 INFO:teuthology.orchestra.run.smithi039.stderr: 1 8 MallocExtension::Register 2023-11-08T12:27:19.122 INFO:teuthology.orchestra.run.smithi039.stderr:----------------------------------------------------- 2023-11-08T12:27:19.122 INFO:teuthology.orchestra.run.smithi039.stderr: 2023-11-08T12:27:19.180 INFO:teuthology.orchestra.run.smithi039.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-08T12:27:19.181 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:19,181 seastar - Reactor backend: linux-aio 2023-11-08T12:27:19.199 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:19,198 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-08T12:27:19.199 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:19,198 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-08T12:27:19.200 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:19,198 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-08T12:27:19.206 INFO:teuthology.orchestra.run.smithi039.stderr:INFO 2023-11-08 12:27:19,206 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-08T12:27:19.209 INFO:teuthology.orchestra.run.smithi039.stderr:==80220==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-08T12:27:19.215 INFO:teuthology.orchestra.run.smithi039.stderr:==80220==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc8d47e000; bottom 0x7f4bb3b7b000; size: 0x00b0d9903000 (759564349440) 2023-11-08T12:27:19.215 INFO:teuthology.orchestra.run.smithi039.stderr:False positive error reports may follow 2023-11-08T12:27:19.215 INFO:teuthology.orchestra.run.smithi039.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-08T12:27:19.241 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0x22b31d69 0x22b399a9 0x22b3ea3f 0x22b440b0 0x24a1c7fb 0x24a234e6 0x24a25e34 0x24a2700d 0x2258a2b7 0x225daf14 0x225fec49 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1eb826c8 0x3feaf 0x3ff5f 0x1e62ba44 2023-11-08T12:27:19.241 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 2023-11-08T12:27:19.303 INFO:teuthology.orchestra.run.smithi039.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x26251dd8 0x26254421 0x26256232 0x26258a97 0x2608d333 0x1ec3ac08 0x1ebb0927 0x1ebb4768 0x2c9f947b 2023-11-08T12:27:19.303 INFO:teuthology.orchestra.run.smithi039.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff93c65af7 0xffffffff93c61447 0xffffffff93e000aa 2023-11-08T12:27:23.272 INFO:teuthology.orchestra.run.smithi039.stderr:----------------------------------------------------- 2023-11-08T12:27:23.272 INFO:teuthology.orchestra.run.smithi039.stderr:Suppressions used: 2023-11-08T12:27:23.272 INFO:teuthology.orchestra.run.smithi039.stderr: count bytes template 2023-11-08T12:27:23.273 INFO:teuthology.orchestra.run.smithi039.stderr: 1 8 MallocExtension::Register 2023-11-08T12:27:23.273 INFO:teuthology.orchestra.run.smithi039.stderr:----------------------------------------------------- 2023-11-08T12:27:23.273 INFO:teuthology.orchestra.run.smithi039.stderr: 2023-11-08T12:27:23.273 INFO:teuthology.orchestra.run.smithi039.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid a1557d23-2394-44bb-8ae3-ddbb181f1a30 2023-11-08T12:27:23.312 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-11-08T12:27:23.344 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-08T12:27:23.344 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-11-08T12:27:23.344 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-11-08T12:27:23.414 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-11-08T12:27:23.415 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-11-08T12:27:23.485 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-11-08T12:27:23.486 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-11-08T12:27:23.556 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-11-08T12:27:23.556 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-11-08T12:27:23.627 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-11-08T12:27:23.627 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-11-08T12:27:23.682 INFO:tasks.ceph:Adding keys to all mons... 2023-11-08T12:27:23.683 DEBUG:teuthology.orchestra.run.smithi039:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-08T12:27:23.751 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr.x] 2023-11-08T12:27:23.752 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCWfktlm8t9FhAA8EsU58yaN1vovs2czSQArg== 2023-11-08T12:27:23.752 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.0] 2023-11-08T12:27:23.752 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCcfktlhT0XIRAAwnQPgdR2L07/zJgPyBWCHQ== 2023-11-08T12:27:23.752 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.1] 2023-11-08T12:27:23.753 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQChfktlpOnxNBAAkYEOpHjmyv14AExDwCX6qQ== 2023-11-08T12:27:23.753 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.2] 2023-11-08T12:27:23.753 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCnfktl6XG6DxAAqOrXNAzfA5hnahZx7lcv3g== 2023-11-08T12:27:23.754 INFO:teuthology.orchestra.run.smithi039.stdout:[client.0] 2023-11-08T12:27:23.754 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCWfktlBy7fGRAA8ovMwPSVd+yin/Z1cG3mcw== 2023-11-08T12:27:23.755 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-08T12:27:23.858 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T12:27:23.921 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T12:27:23.984 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-08T12:27:24.046 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-08T12:27:24.111 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-08T12:27:24.112 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-11-08T12:27:24.143 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-08T12:27:24.250 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:24.248+0000 7f672ea24980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:24.250 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:24.248+0000 7f672ea24980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:24.272 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-11-08T12:27:24.303 DEBUG:teuthology.orchestra.run.smithi039:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-08T12:27:24.359 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-11-08T12:27:24.359 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-08T12:27:24.360 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-11-08T12:27:24.401 INFO:tasks.ceph.mon.a:Started 2023-11-08T12:27:24.402 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-11-08T12:27:24.402 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-11-08T12:27:24.403 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-11-08T12:27:24.405 INFO:tasks.ceph.mgr.x:Started 2023-11-08T12:27:24.405 INFO:tasks.ceph:Setting crush tunables to default 2023-11-08T12:27:24.406 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd crush tunables default 2023-11-08T12:27:24.488 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-11-08T12:27:24.486+0000 7fb3f8435980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:24.488 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-11-08T12:27:24.486+0000 7fb3f8435980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:24.488 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-11-08T12:27:24.486+0000 7fb3f8435980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:24.489 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:24.488+0000 7f6356617f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:24.520 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:24.513+0000 7fd5e402d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:24.547 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:24.545+0000 7fd5e402d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:24.617 INFO:teuthology.orchestra.run.smithi039.stderr:adjusted tunables profile to default 2023-11-08T12:27:24.628 INFO:tasks.ceph:check_enable_crimson: True 2023-11-08T12:27:24.628 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2023-11-08T12:27:24.628 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2023-11-08T12:27:24.693 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:24.691+0000 7f6356617f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:24.693 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:24.691+0000 7f6356617f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:24.738 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:24.732+0000 7efe6a11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:24.740 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:24.739+0000 7efe6a11e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:24.823 INFO:tasks.ceph.mon.a.smithi039.stderr:2023-11-08T12:27:24.821+0000 7fb3ef7f1640 -1 WARNING: experimental feature 'crimson' is enabled 2023-11-08T12:27:24.824 INFO:tasks.ceph.mon.a.smithi039.stderr:Please be aware that this feature is experimental, untested, 2023-11-08T12:27:24.824 INFO:tasks.ceph.mon.a.smithi039.stderr:unsupported, and may result in data corruption, data loss, 2023-11-08T12:27:24.824 INFO:tasks.ceph.mon.a.smithi039.stderr:and/or irreparable damage to your cluster. Do not use 2023-11-08T12:27:24.824 INFO:tasks.ceph.mon.a.smithi039.stderr:feature with important data. 2023-11-08T12:27:24.825 INFO:tasks.ceph.mon.a.smithi039.stderr: 2023-11-08T12:27:25.182 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:25.180+0000 7f6356617f00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-08T12:27:25.235 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:25.233+0000 7f6356617f00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-08T12:27:25.630 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-11-08T12:27:25.630 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-11-08T12:27:25.631 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-11-08T12:27:25.662 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-11-08T12:27:25.662 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-11-08T12:27:25.734 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-11-08T12:27:25.734 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-11-08T12:27:25.804 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:25.802+0000 7f6356617f00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-08T12:27:25.806 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd new 9e6c432a-f33c-4a6e-a8ca-1dffecfcc0e8 0 2023-11-08T12:27:25.859 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:25.857+0000 7f6356617f00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-08T12:27:25.907 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:25.905+0000 7f6356617f00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-08T12:27:25.957 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:25.950+0000 7fe0a0ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:25.958 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:25.957+0000 7fe0a0ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:25.968 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:25.966+0000 7f6356617f00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-08T12:27:26.036 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:26.034+0000 7f6356617f00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-08T12:27:26.073 INFO:teuthology.orchestra.run.smithi039.stdout:0 2023-11-08T12:27:26.084 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd new c1423571-3cca-4144-8783-04c7c37e1862 1 2023-11-08T12:27:26.144 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:26.142+0000 7f6356617f00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-08T12:27:26.220 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:26.219+0000 7f6356617f00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-08T12:27:26.236 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:26.230+0000 7f58be99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:26.238 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:26.238+0000 7f58be99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:26.288 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:26.287+0000 7f6356617f00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-08T12:27:26.347 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:26.346+0000 7f6356617f00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-08T12:27:26.348 INFO:teuthology.orchestra.run.smithi039.stdout:1 2023-11-08T12:27:26.359 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd new b248f978-d833-4cb2-a690-5c1dc6961f8b 2 2023-11-08T12:27:26.398 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:26.397+0000 7f6356617f00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-08T12:27:26.508 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:26.502+0000 7f98f642d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:26.510 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:26.510+0000 7f98f642d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:26.591 INFO:teuthology.orchestra.run.smithi039.stdout:2 2023-11-08T12:27:26.602 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2023-11-08T12:27:26.639 INFO:tasks.ceph.osd.0:Restarting daemon 2023-11-08T12:27:26.639 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2023-11-08T12:27:26.681 INFO:tasks.ceph.osd.0:Started 2023-11-08T12:27:26.681 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2023-11-08T12:27:26.719 INFO:tasks.ceph.osd.1:Restarting daemon 2023-11-08T12:27:26.719 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2023-11-08T12:27:26.761 INFO:tasks.ceph.osd.1:Started 2023-11-08T12:27:26.761 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2023-11-08T12:27:26.800 INFO:tasks.ceph.osd.2:Restarting daemon 2023-11-08T12:27:26.800 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2023-11-08T12:27:26.818 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:26.817+0000 7f6356617f00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-08T12:27:26.820 INFO:tasks.ceph.osd.2:Started 2023-11-08T12:27:26.821 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T12:27:27.026 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:27.017+0000 7f556946d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:27.029 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:27.029+0000 7f556946d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:27.046 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:27.045+0000 7f6356617f00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-08T12:27:27.169 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:27.169 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":6,"fsid":"a1557d23-2394-44bb-8ae3-ddbb181f1a30","created":"2023-11-08T12:27:24.530116+0000","modified":"2023-11-08T12:27:26.588179+0000","last_up_change":"0.000000","last_in_change":"2023-11-08T12:27:26.588179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"9e6c432a-f33c-4a6e-a8ca-1dffecfcc0e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c1423571-3cca-4144-8783-04c7c37e1862","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b248f978-d833-4cb2-a690-5c1dc6961f8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T12:27:27.188 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:27.187+0000 7f6356617f00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-08T12:27:27.192 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-11-08T12:27:27.193 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-08T12:27:27.245 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:27.244+0000 7f6356617f00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-08T12:27:27.355 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:27.354+0000 7f6356617f00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-08T12:27:27.457 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:27.456+0000 7f6356617f00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-08T12:27:27.496 DEBUG:teuthology.orchestra.run.smithi039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-08T12:27:27.595 INFO:teuthology.misc.health.smithi039.stderr:2023-11-08T12:27:27.592+0000 7f7f02d08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:27.598 INFO:teuthology.misc.health.smithi039.stderr:2023-11-08T12:27:27.597+0000 7f7f02d08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:27.608 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:27.608+0000 7f6356617f00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-08T12:27:27.657 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:27.656+0000 7f6356617f00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-08T12:27:27.665 INFO:teuthology.misc.health.smithi039.stdout: 2023-11-08T12:27:27.665 INFO:teuthology.misc.health.smithi039.stdout:{"epoch":6,"fsid":"a1557d23-2394-44bb-8ae3-ddbb181f1a30","created":"2023-11-08T12:27:24.530116+0000","modified":"2023-11-08T12:27:26.588179+0000","last_up_change":"0.000000","last_in_change":"2023-11-08T12:27:26.588179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"9e6c432a-f33c-4a6e-a8ca-1dffecfcc0e8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c1423571-3cca-4144-8783-04c7c37e1862","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b248f978-d833-4cb2-a690-5c1dc6961f8b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T12:27:27.676 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-11-08T12:27:27.814 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:27.813+0000 7f6356617f00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-08T12:27:28.674 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:28.673+0000 7f6356617f00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-08T12:27:28.960 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:28.959+0000 7f6356617f00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-08T12:27:29.015 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:29.014+0000 7f6356617f00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-08T12:27:29.122 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:29.121+0000 7f6356617f00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-08T12:27:30.142 INFO:tasks.ceph.mgr.x.smithi039.stderr:2023-11-08T12:27:30.141+0000 7f62f686e640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-11-08T12:27:33.979 DEBUG:teuthology.orchestra.run.smithi039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-08T12:27:34.090 INFO:teuthology.misc.health.smithi039.stderr:2023-11-08T12:27:34.086+0000 7f90ca0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:34.092 INFO:teuthology.misc.health.smithi039.stderr:2023-11-08T12:27:34.092+0000 7f90ca0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:34.300 INFO:teuthology.misc.health.smithi039.stdout: 2023-11-08T12:27:34.301 INFO:teuthology.misc.health.smithi039.stdout:{"epoch":9,"fsid":"a1557d23-2394-44bb-8ae3-ddbb181f1a30","created":"2023-11-08T12:27:24.530116+0000","modified":"2023-11-08T12:27:34.161437+0000","last_up_change":"2023-11-08T12:27:32.150807+0000","last_in_change":"2023-11-08T12:27:26.588179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-08T12:27:33.195725+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e6c432a-f33c-4a6e-a8ca-1dffecfcc0e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":2915244434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6803","nonce":2915244434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2915244434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6809","nonce":2915244434}]},"public_addr":"172.21.15.39:6802/2915244434","cluster_addr":"172.21.15.39:6803/2915244434","heartbeat_back_addr":"172.21.15.39:6808/2915244434","heartbeat_front_addr":"172.21.15.39:6809/2915244434","state":["exists","up"]},{"osd":1,"uuid":"c1423571-3cca-4144-8783-04c7c37e1862","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":238952653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":238952653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6811","nonce":238952653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":238952653}]},"public_addr":"172.21.15.39:6804/238952653","cluster_addr":"172.21.15.39:6805/238952653","heartbeat_back_addr":"172.21.15.39:6811/238952653","heartbeat_front_addr":"172.21.15.39:6810/238952653","state":["exists","up"]},{"osd":2,"uuid":"b248f978-d833-4cb2-a690-5c1dc6961f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":338940800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6807","nonce":338940800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6813","nonce":338940800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":338940800}]},"public_addr":"172.21.15.39:6806/338940800","cluster_addr":"172.21.15.39:6807/338940800","heartbeat_back_addr":"172.21.15.39:6813/338940800","heartbeat_front_addr":"172.21.15.39:6812/338940800","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T12:27:34.311 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-11-08T12:27:34.312 INFO:tasks.ceph:Creating RBD pool 2023-11-08T12:27:34.312 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd pool create rbd 8 2023-11-08T12:27:34.423 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:34.416+0000 7f8213a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:34.425 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:34.424+0000 7f8213a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:35.175 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'rbd' created 2023-11-08T12:27:35.190 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-11-08T12:27:35.323 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:35.313+0000 7fa8ac86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:35.325 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:35.325+0000 7fa8ac86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:36.186 INFO:teuthology.orchestra.run.smithi039.stderr:enabled application 'rbd' on pool 'rbd' 2023-11-08T12:27:36.200 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-11-08T12:27:36.201 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-08T12:27:36.201 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-11-08T12:27:36.202 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-11-08T12:27:36.202 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-08T12:27:36.337 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:36.333+0000 7fc44075e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:36.339 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:36.338+0000 7fc44075e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:36.572 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:36.583 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":1052946712},{"type":"v1","addr":"172.21.15.39:6801","nonce":1052946712}]},"active_addr":"172.21.15.39:6801/1052946712","active_change":"2023-11-08T12:27:29.134454+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:v0.0.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":1928025942}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":889791190}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":1317834124}]}]} 2023-11-08T12:27:36.585 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-11-08T12:27:36.586 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-11-08T12:27:36.586 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T12:27:36.698 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:36.695+0000 7ff82eead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:36.700 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:36.700+0000 7ff82eead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:36.885 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:36.885 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":11,"fsid":"a1557d23-2394-44bb-8ae3-ddbb181f1a30","created":"2023-11-08T12:27:24.530116+0000","modified":"2023-11-08T12:27:36.185141+0000","last_up_change":"2023-11-08T12:27:32.150807+0000","last_in_change":"2023-11-08T12:27:26.588179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-08T12:27:33.195725+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-08T12:27:34.639378+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e6c432a-f33c-4a6e-a8ca-1dffecfcc0e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":2915244434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6803","nonce":2915244434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2915244434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6809","nonce":2915244434}]},"public_addr":"172.21.15.39:6802/2915244434","cluster_addr":"172.21.15.39:6803/2915244434","heartbeat_back_addr":"172.21.15.39:6808/2915244434","heartbeat_front_addr":"172.21.15.39:6809/2915244434","state":["exists","up"]},{"osd":1,"uuid":"c1423571-3cca-4144-8783-04c7c37e1862","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":238952653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":238952653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6811","nonce":238952653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":238952653}]},"public_addr":"172.21.15.39:6804/238952653","cluster_addr":"172.21.15.39:6805/238952653","heartbeat_back_addr":"172.21.15.39:6811/238952653","heartbeat_front_addr":"172.21.15.39:6810/238952653","state":["exists","up"]},{"osd":2,"uuid":"b248f978-d833-4cb2-a690-5c1dc6961f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":338940800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6807","nonce":338940800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6813","nonce":338940800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":338940800}]},"public_addr":"172.21.15.39:6806/338940800","cluster_addr":"172.21.15.39:6807/338940800","heartbeat_back_addr":"172.21.15.39:6813/338940800","heartbeat_front_addr":"172.21.15.39:6812/338940800","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T12:27:36.897 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-11-08T12:27:36.897 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-08T12:27:37.009 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:37.005+0000 7fcbb802d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:37.011 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:37.010+0000 7fcbb802d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:37.220 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:37.220 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":12,"fsid":"a1557d23-2394-44bb-8ae3-ddbb181f1a30","created":"2023-11-08T12:27:24.530116+0000","modified":"2023-11-08T12:27:37.193000+0000","last_up_change":"2023-11-08T12:27:32.150807+0000","last_in_change":"2023-11-08T12:27:26.588179+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-08T12:27:33.195725+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-08T12:27:34.639378+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9e6c432a-f33c-4a6e-a8ca-1dffecfcc0e8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":2915244434}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6803","nonce":2915244434}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":2915244434}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6809","nonce":2915244434}]},"public_addr":"172.21.15.39:6802/2915244434","cluster_addr":"172.21.15.39:6803/2915244434","heartbeat_back_addr":"172.21.15.39:6808/2915244434","heartbeat_front_addr":"172.21.15.39:6809/2915244434","state":["exists","up"]},{"osd":1,"uuid":"c1423571-3cca-4144-8783-04c7c37e1862","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":238952653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":238952653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6811","nonce":238952653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":238952653}]},"public_addr":"172.21.15.39:6804/238952653","cluster_addr":"172.21.15.39:6805/238952653","heartbeat_back_addr":"172.21.15.39:6811/238952653","heartbeat_front_addr":"172.21.15.39:6810/238952653","state":["exists","up"]},{"osd":2,"uuid":"b248f978-d833-4cb2-a690-5c1dc6961f8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":338940800}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6807","nonce":338940800}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6813","nonce":338940800}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":338940800}]},"public_addr":"172.21.15.39:6806/338940800","cluster_addr":"172.21.15.39:6807/338940800","heartbeat_back_addr":"172.21.15.39:6813/338940800","heartbeat_front_addr":"172.21.15.39:6812/338940800","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-08T12:27:37.234 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-11-08T12:27:37.235 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-11-08T12:27:37.235 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-11-08T12:27:37.354 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:37.349+0000 7fc77cb1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:37.354 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:37.349+0000 7f378606d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:37.355 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:37.349+0000 7fecece2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:37.356 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:37.355+0000 7f378606d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:37.356 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:37.355+0000 7fc77cb1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:37.357 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:37.356+0000 7fecece2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:37.382 INFO:teuthology.orchestra.run.smithi039.stdout:34359738373 2023-11-08T12:27:37.382 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-08T12:27:37.384 INFO:teuthology.orchestra.run.smithi039.stdout:34359738373 2023-11-08T12:27:37.384 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-08T12:27:37.385 INFO:teuthology.orchestra.run.smithi039.stdout:34359738373 2023-11-08T12:27:37.386 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-08T12:27:37.502 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:37.497+0000 7ff90739e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:37.503 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:37.497+0000 7f03f8c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:37.503 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:37.497+0000 7f2d4575e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:37.505 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:37.504+0000 7ff90739e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:37.505 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:37.504+0000 7f2d4575e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:37.506 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:37.504+0000 7f03f8c2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:37.693 INFO:teuthology.orchestra.run.smithi039.stdout:34359738373 2023-11-08T12:27:37.697 INFO:teuthology.orchestra.run.smithi039.stdout:34359738373 2023-11-08T12:27:37.699 INFO:teuthology.orchestra.run.smithi039.stdout:34359738373 2023-11-08T12:27:37.707 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2023-11-08T12:27:37.708 DEBUG:teuthology.parallel:result is None 2023-11-08T12:27:37.711 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2023-11-08T12:27:37.712 DEBUG:teuthology.parallel:result is None 2023-11-08T12:27:37.713 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2023-11-08T12:27:37.713 DEBUG:teuthology.parallel:result is None 2023-11-08T12:27:37.714 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-08T12:27:37.714 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-08T12:27:37.866 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:37.862+0000 7f62a3b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:37.868 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:37.867+0000 7f62a3b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:38.050 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:38.051 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2023-11-08T12:27:38.062 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-11-08T12:27:37.198819+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80760,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034760,"statfs":{"total":289910292480,"available":289827594240,"internally_reserved":0,"allocated":270336,"data_stored":1224791,"data_compressed":60098,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.006145"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:27:36.286800+0000","last_change":"2023-11-08T12:27:36.286857+0000","last_active":"2023-11-08T12:27:36.286800+0000","last_peered":"2023-11-08T12:27:36.286800+0000","last_clean":"2023-11-08T12:27:35.173869+0000","last_became_active":"2023-11-08T12:27:36.285111+0000","last_became_peered":"2023-11-08T12:27:36.285111+0000","last_unstale":"2023-11-08T12:27:36.286800+0000","last_undegraded":"2023-11-08T12:27:36.286800+0000","last_fullsized":"2023-11-08T12:27:36.286800+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_clean_scrub_stamp":"2023-11-08T12:27:35.173869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:27:36.299960+0000","last_change":"2023-11-08T12:27:36.300048+0000","last_active":"2023-11-08T12:27:36.299960+0000","last_peered":"2023-11-08T12:27:36.299960+0000","last_clean":"2023-11-08T12:27:35.173869+0000","last_became_active":"2023-11-08T12:27:36.297523+0000","last_became_peered":"2023-11-08T12:27:36.297523+0000","last_unstale":"2023-11-08T12:27:36.299960+0000","last_undegraded":"2023-11-08T12:27:36.299960+0000","last_fullsized":"2023-11-08T12:27:36.299960+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_clean_scrub_stamp":"2023-11-08T12:27:35.173869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:27:36.306491+0000","last_change":"2023-11-08T12:27:36.306583+0000","last_active":"2023-11-08T12:27:36.306491+0000","last_peered":"2023-11-08T12:27:36.306491+0000","last_clean":"2023-11-08T12:27:35.173869+0000","last_became_active":"2023-11-08T12:27:36.304105+0000","last_became_peered":"2023-11-08T12:27:36.304105+0000","last_unstale":"2023-11-08T12:27:36.306491+0000","last_undegraded":"2023-11-08T12:27:36.306491+0000","last_fullsized":"2023-11-08T12:27:36.306491+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_clean_scrub_stamp":"2023-11-08T12:27:35.173869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:27:36.267081+0000","last_change":"2023-11-08T12:27:36.267191+0000","last_active":"2023-11-08T12:27:36.267081+0000","last_peered":"2023-11-08T12:27:36.267081+0000","last_clean":"2023-11-08T12:27:35.173869+0000","last_became_active":"2023-11-08T12:27:36.259307+0000","last_became_peered":"2023-11-08T12:27:36.259307+0000","last_unstale":"2023-11-08T12:27:36.267081+0000","last_undegraded":"2023-11-08T12:27:36.267081+0000","last_fullsized":"2023-11-08T12:27:36.267081+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_clean_scrub_stamp":"2023-11-08T12:27:35.173869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:27:36.270513+0000","last_change":"2023-11-08T12:27:36.270599+0000","last_active":"2023-11-08T12:27:36.270513+0000","last_peered":"2023-11-08T12:27:36.270513+0000","last_clean":"2023-11-08T12:27:35.173869+0000","last_became_active":"2023-11-08T12:27:36.267824+0000","last_became_peered":"2023-11-08T12:27:36.267824+0000","last_unstale":"2023-11-08T12:27:36.270513+0000","last_undegraded":"2023-11-08T12:27:36.270513+0000","last_fullsized":"2023-11-08T12:27:36.270513+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_clean_scrub_stamp":"2023-11-08T12:27:35.173869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:27:36.286830+0000","last_change":"2023-11-08T12:27:36.286892+0000","last_active":"2023-11-08T12:27:36.286830+0000","last_peered":"2023-11-08T12:27:36.286830+0000","last_clean":"2023-11-08T12:27:35.173869+0000","last_became_active":"2023-11-08T12:27:36.284636+0000","last_became_peered":"2023-11-08T12:27:36.284636+0000","last_unstale":"2023-11-08T12:27:36.286830+0000","last_undegraded":"2023-11-08T12:27:36.286830+0000","last_fullsized":"2023-11-08T12:27:36.286830+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_clean_scrub_stamp":"2023-11-08T12:27:35.173869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:27:36.277763+0000","last_change":"2023-11-08T12:27:36.277826+0000","last_active":"2023-11-08T12:27:36.277763+0000","last_peered":"2023-11-08T12:27:36.277763+0000","last_clean":"2023-11-08T12:27:35.173869+0000","last_became_active":"2023-11-08T12:27:36.275041+0000","last_became_peered":"2023-11-08T12:27:36.275041+0000","last_unstale":"2023-11-08T12:27:36.277763+0000","last_undegraded":"2023-11-08T12:27:36.277763+0000","last_fullsized":"2023-11-08T12:27:36.277763+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_clean_scrub_stamp":"2023-11-08T12:27:35.173869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:27:36.264792+0000","last_change":"2023-11-08T12:27:36.264879+0000","last_active":"2023-11-08T12:27:36.264792+0000","last_peered":"2023-11-08T12:27:36.264792+0000","last_clean":"2023-11-08T12:27:35.173869+0000","last_became_active":"2023-11-08T12:27:36.262655+0000","last_became_peered":"2023-11-08T12:27:36.262655+0000","last_unstale":"2023-11-08T12:27:36.264792+0000","last_undegraded":"2023-11-08T12:27:36.264792+0000","last_fullsized":"2023-11-08T12:27:36.264792+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_clean_scrub_stamp":"2023-11-08T12:27:35.173869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:27:36.211505+0000","last_change":"2023-11-08T12:27:35.242878+0000","last_active":"2023-11-08T12:27:36.211505+0000","last_peered":"2023-11-08T12:27:36.211505+0000","last_clean":"2023-11-08T12:27:36.211505+0000","last_became_active":"2023-11-08T12:27:35.232295+0000","last_became_peered":"2023-11-08T12:27:35.232295+0000","last_unstale":"2023-11-08T12:27:36.211505+0000","last_undegraded":"2023-11-08T12:27:36.211505+0000","last_fullsized":"2023-11-08T12:27:36.211505+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:27:34.161437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:27:34.161437+0000","last_clean_scrub_stamp":"2023-11-08T12:27:34.161437+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26872,"kb_used_data":48,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344968,"statfs":{"total":96636764160,"available":96609247232,"internally_reserved":0,"allocated":49152,"data_stored":14685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-08T12:27:38.063 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-08T12:27:38.176 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:38.172+0000 7f0f8f464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:38.178 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:38.177+0000 7f0f8f464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:38.367 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:38.367 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2023-11-08T12:27:38.379 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-11-08T12:27:37.198819+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80760,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034760,"statfs":{"total":289910292480,"available":289827594240,"internally_reserved":0,"allocated":270336,"data_stored":1224791,"data_compressed":60098,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.006145"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:27:36.286800+0000","last_change":"2023-11-08T12:27:36.286857+0000","last_active":"2023-11-08T12:27:36.286800+0000","last_peered":"2023-11-08T12:27:36.286800+0000","last_clean":"2023-11-08T12:27:35.173869+0000","last_became_active":"2023-11-08T12:27:36.285111+0000","last_became_peered":"2023-11-08T12:27:36.285111+0000","last_unstale":"2023-11-08T12:27:36.286800+0000","last_undegraded":"2023-11-08T12:27:36.286800+0000","last_fullsized":"2023-11-08T12:27:36.286800+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_clean_scrub_stamp":"2023-11-08T12:27:35.173869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:27:36.299960+0000","last_change":"2023-11-08T12:27:36.300048+0000","last_active":"2023-11-08T12:27:36.299960+0000","last_peered":"2023-11-08T12:27:36.299960+0000","last_clean":"2023-11-08T12:27:35.173869+0000","last_became_active":"2023-11-08T12:27:36.297523+0000","last_became_peered":"2023-11-08T12:27:36.297523+0000","last_unstale":"2023-11-08T12:27:36.299960+0000","last_undegraded":"2023-11-08T12:27:36.299960+0000","last_fullsized":"2023-11-08T12:27:36.299960+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_clean_scrub_stamp":"2023-11-08T12:27:35.173869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:27:36.306491+0000","last_change":"2023-11-08T12:27:36.306583+0000","last_active":"2023-11-08T12:27:36.306491+0000","last_peered":"2023-11-08T12:27:36.306491+0000","last_clean":"2023-11-08T12:27:35.173869+0000","last_became_active":"2023-11-08T12:27:36.304105+0000","last_became_peered":"2023-11-08T12:27:36.304105+0000","last_unstale":"2023-11-08T12:27:36.306491+0000","last_undegraded":"2023-11-08T12:27:36.306491+0000","last_fullsized":"2023-11-08T12:27:36.306491+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_clean_scrub_stamp":"2023-11-08T12:27:35.173869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:27:36.267081+0000","last_change":"2023-11-08T12:27:36.267191+0000","last_active":"2023-11-08T12:27:36.267081+0000","last_peered":"2023-11-08T12:27:36.267081+0000","last_clean":"2023-11-08T12:27:35.173869+0000","last_became_active":"2023-11-08T12:27:36.259307+0000","last_became_peered":"2023-11-08T12:27:36.259307+0000","last_unstale":"2023-11-08T12:27:36.267081+0000","last_undegraded":"2023-11-08T12:27:36.267081+0000","last_fullsized":"2023-11-08T12:27:36.267081+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_clean_scrub_stamp":"2023-11-08T12:27:35.173869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:27:36.270513+0000","last_change":"2023-11-08T12:27:36.270599+0000","last_active":"2023-11-08T12:27:36.270513+0000","last_peered":"2023-11-08T12:27:36.270513+0000","last_clean":"2023-11-08T12:27:35.173869+0000","last_became_active":"2023-11-08T12:27:36.267824+0000","last_became_peered":"2023-11-08T12:27:36.267824+0000","last_unstale":"2023-11-08T12:27:36.270513+0000","last_undegraded":"2023-11-08T12:27:36.270513+0000","last_fullsized":"2023-11-08T12:27:36.270513+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_clean_scrub_stamp":"2023-11-08T12:27:35.173869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:27:36.286830+0000","last_change":"2023-11-08T12:27:36.286892+0000","last_active":"2023-11-08T12:27:36.286830+0000","last_peered":"2023-11-08T12:27:36.286830+0000","last_clean":"2023-11-08T12:27:35.173869+0000","last_became_active":"2023-11-08T12:27:36.284636+0000","last_became_peered":"2023-11-08T12:27:36.284636+0000","last_unstale":"2023-11-08T12:27:36.286830+0000","last_undegraded":"2023-11-08T12:27:36.286830+0000","last_fullsized":"2023-11-08T12:27:36.286830+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_clean_scrub_stamp":"2023-11-08T12:27:35.173869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:27:36.277763+0000","last_change":"2023-11-08T12:27:36.277826+0000","last_active":"2023-11-08T12:27:36.277763+0000","last_peered":"2023-11-08T12:27:36.277763+0000","last_clean":"2023-11-08T12:27:35.173869+0000","last_became_active":"2023-11-08T12:27:36.275041+0000","last_became_peered":"2023-11-08T12:27:36.275041+0000","last_unstale":"2023-11-08T12:27:36.277763+0000","last_undegraded":"2023-11-08T12:27:36.277763+0000","last_fullsized":"2023-11-08T12:27:36.277763+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_clean_scrub_stamp":"2023-11-08T12:27:35.173869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:27:36.264792+0000","last_change":"2023-11-08T12:27:36.264879+0000","last_active":"2023-11-08T12:27:36.264792+0000","last_peered":"2023-11-08T12:27:36.264792+0000","last_clean":"2023-11-08T12:27:35.173869+0000","last_became_active":"2023-11-08T12:27:36.262655+0000","last_became_peered":"2023-11-08T12:27:36.262655+0000","last_unstale":"2023-11-08T12:27:36.264792+0000","last_undegraded":"2023-11-08T12:27:36.264792+0000","last_fullsized":"2023-11-08T12:27:36.264792+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:27:35.173869+0000","last_clean_scrub_stamp":"2023-11-08T12:27:35.173869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-11-08T12:27:36.211505+0000","last_change":"2023-11-08T12:27:35.242878+0000","last_active":"2023-11-08T12:27:36.211505+0000","last_peered":"2023-11-08T12:27:36.211505+0000","last_clean":"2023-11-08T12:27:36.211505+0000","last_became_active":"2023-11-08T12:27:35.232295+0000","last_became_peered":"2023-11-08T12:27:35.232295+0000","last_unstale":"2023-11-08T12:27:36.211505+0000","last_undegraded":"2023-11-08T12:27:36.211505+0000","last_fullsized":"2023-11-08T12:27:36.211505+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-08T12:27:34.161437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-08T12:27:34.161437+0000","last_clean_scrub_stamp":"2023-11-08T12:27:34.161437+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26872,"kb_used_data":48,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344968,"statfs":{"total":96636764160,"available":96609247232,"internally_reserved":0,"allocated":49152,"data_stored":14685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605053,"data_compressed":30049,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-08T12:27:38.380 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-08T12:27:38.380 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-08T12:27:38.380 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-11-08T12:27:38.381 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-08T12:27:38.491 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:38.488+0000 7ff796f08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:38.493 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:27:38.492+0000 7ff796f08640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:27:38.711 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:38.712 INFO:teuthology.orchestra.run.smithi039.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-08T12:27:38.724 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-11-08T12:27:38.724 INFO:teuthology.run_tasks:Running task cbt... 2023-11-08T12:27:38.738 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi039.front.sepia.ceph.com', 'osds': ['ubuntu@smithi039.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi039.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi039.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi039.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2023-11-08T12:27:38.738 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2023-11-08T12:27:38.756 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-11-08T12:27:38.757 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-11-08T12:27:38.808 DEBUG:teuthology.orchestra.run.smithi039:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2023-11-08T12:27:38.864 INFO:teuthology.orchestra.run.smithi039.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2023-11-08T12:27:39.258 INFO:tasks.cbt:Installing collectl 2023-11-08T12:27:39.259 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2023-11-08T12:27:39.365 INFO:teuthology.orchestra.run.smithi039.stderr:--2023-11-08 12:27:39-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2023-11-08T12:27:39.509 INFO:teuthology.orchestra.run.smithi039.stderr:Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::6812:256f, ... 2023-11-08T12:27:39.520 INFO:teuthology.orchestra.run.smithi039.stderr:Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. 2023-11-08T12:27:39.844 INFO:teuthology.orchestra.run.smithi039.stderr:HTTP request sent, awaiting response... 302 Found 2023-11-08T12:27:39.844 INFO:teuthology.orchestra.run.smithi039.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlS3672yIoF0qun1pN4ohRSiIK-vNdmq8LFIHyNyo64_UKBePxueW0g-N4NSHxTUuffWok_YcTkEzIUH0veJJzTn7HGg%3D%3D&use_mirror=cfhcable&r= [following] 2023-11-08T12:27:39.844 INFO:teuthology.orchestra.run.smithi039.stderr:--2023-11-08 12:27:39-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlS3672yIoF0qun1pN4ohRSiIK-vNdmq8LFIHyNyo64_UKBePxueW0g-N4NSHxTUuffWok_YcTkEzIUH0veJJzTn7HGg%3D%3D&use_mirror=cfhcable&r= 2023-11-08T12:27:39.854 INFO:teuthology.orchestra.run.smithi039.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2023-11-08T12:27:39.926 INFO:teuthology.orchestra.run.smithi039.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2023-11-08T12:27:40.162 INFO:teuthology.orchestra.run.smithi039.stderr:HTTP request sent, awaiting response... 302 Found 2023-11-08T12:27:40.162 INFO:teuthology.orchestra.run.smithi039.stderr:Location: https://cfhcable.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2023-11-08T12:27:40.162 INFO:teuthology.orchestra.run.smithi039.stderr:--2023-11-08 12:27:40-- https://cfhcable.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2023-11-08T12:27:40.172 INFO:teuthology.orchestra.run.smithi039.stderr:Resolving cfhcable.dl.sourceforge.net (cfhcable.dl.sourceforge.net)... 146.71.73.6 2023-11-08T12:27:40.207 INFO:teuthology.orchestra.run.smithi039.stderr:Connecting to cfhcable.dl.sourceforge.net (cfhcable.dl.sourceforge.net)|146.71.73.6|:443... connected. 2023-11-08T12:27:40.359 INFO:teuthology.orchestra.run.smithi039.stderr:HTTP request sent, awaiting response... 200 OK 2023-11-08T12:27:40.359 INFO:teuthology.orchestra.run.smithi039.stderr:Length: 620693 (606K) [application/x-gzip] 2023-11-08T12:27:40.359 INFO:teuthology.orchestra.run.smithi039.stderr:Saving to: ‘collectl.tar.gz’ 2023-11-08T12:27:40.359 INFO:teuthology.orchestra.run.smithi039.stderr: 2023-11-08T12:27:40.394 INFO:teuthology.orchestra.run.smithi039.stderr: 0K .......... .......... .......... .......... .......... 8% 1.35M 0s 2023-11-08T12:27:40.429 INFO:teuthology.orchestra.run.smithi039.stderr: 50K .......... .......... .......... .......... .......... 16% 1.43M 0s 2023-11-08T12:27:40.430 INFO:teuthology.orchestra.run.smithi039.stderr: 100K .......... .......... .......... .......... .......... 24% 28.8M 0s 2023-11-08T12:27:40.470 INFO:teuthology.orchestra.run.smithi039.stderr: 150K .......... .......... .......... .......... .......... 32% 1.25M 0s 2023-11-08T12:27:40.470 INFO:teuthology.orchestra.run.smithi039.stderr: 200K .......... .......... .......... .......... .......... 41% 98.2M 0s 2023-11-08T12:27:40.471 INFO:teuthology.orchestra.run.smithi039.stderr: 250K .......... .......... .......... .......... .......... 49% 28.0M 0s 2023-11-08T12:27:40.492 INFO:teuthology.orchestra.run.smithi039.stderr: 300K .......... .......... .......... .......... .......... 57% 2.39M 0s 2023-11-08T12:27:40.518 INFO:teuthology.orchestra.run.smithi039.stderr: 350K .......... .......... .......... .......... .......... 65% 1.99M 0s 2023-11-08T12:27:40.518 INFO:teuthology.orchestra.run.smithi039.stderr: 400K .......... .......... .......... .......... .......... 74% 92.7M 0s 2023-11-08T12:27:40.519 INFO:teuthology.orchestra.run.smithi039.stderr: 450K .......... .......... .......... .......... .......... 82% 36.7M 0s 2023-11-08T12:27:40.534 INFO:teuthology.orchestra.run.smithi039.stderr: 500K .......... .......... .......... .......... .......... 90% 3.20M 0s 2023-11-08T12:27:40.535 INFO:teuthology.orchestra.run.smithi039.stderr: 550K .......... .......... .......... .......... .......... 98% 42.5M 0s 2023-11-08T12:27:40.535 INFO:teuthology.orchestra.run.smithi039.stderr: 600K ...... 100% 11.4T=0.2s 2023-11-08T12:27:40.535 INFO:teuthology.orchestra.run.smithi039.stderr: 2023-11-08T12:27:40.535 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08 12:27:40 (3.35 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2023-11-08T12:27:40.536 INFO:teuthology.orchestra.run.smithi039.stderr: 2023-11-08T12:27:40.576 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/ 2023-11-08T12:27:40.576 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/collectl.conf 2023-11-08T12:27:40.576 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/ 2023-11-08T12:27:40.576 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/SlowProc.html 2023-11-08T12:27:40.577 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/BuddyInfo.html 2023-11-08T12:27:40.577 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/WhySummary.html 2023-11-08T12:27:40.577 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Socket.html 2023-11-08T12:27:40.577 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/colmux.html 2023-11-08T12:27:40.577 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Tutorial.html 2023-11-08T12:27:40.578 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Gexpr.html 2023-11-08T12:27:40.578 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Import.html 2023-11-08T12:27:40.578 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/style.css 2023-11-08T12:27:40.578 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2023-11-08T12:27:40.578 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2023-11-08T12:27:40.579 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/FileNaming.html 2023-11-08T12:27:40.579 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/TheMath.html 2023-11-08T12:27:40.579 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Environmental.html 2023-11-08T12:27:40.580 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Process.html 2023-11-08T12:27:40.580 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/OutputFiles.html 2023-11-08T12:27:40.580 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/HiResTime.html 2023-11-08T12:27:40.580 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/OutputFormats.html 2023-11-08T12:27:40.580 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/slub.jpg 2023-11-08T12:27:40.581 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Lustre.html 2023-11-08T12:27:40.581 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Documentation.html 2023-11-08T12:27:40.581 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Inodes.html 2023-11-08T12:27:40.581 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Graphite.html 2023-11-08T12:27:40.581 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/InputFiles.html 2023-11-08T12:27:40.581 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Export.html 2023-11-08T12:27:40.582 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Interrupts.html 2023-11-08T12:27:40.582 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Startup.html 2023-11-08T12:27:40.582 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Performance.html 2023-11-08T12:27:40.582 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Messages.html 2023-11-08T12:27:40.582 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Logging.html 2023-11-08T12:27:40.583 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Data-detail.html 2023-11-08T12:27:40.583 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/CPUs.html 2023-11-08T12:27:40.583 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Matrix.html 2023-11-08T12:27:40.583 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Data-brief.html 2023-11-08T12:27:40.583 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/SlabInfo.html 2023-11-08T12:27:40.583 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Misc.html 2023-11-08T12:27:40.584 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Examples.html 2023-11-08T12:27:40.584 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/OpenStack.html 2023-11-08T12:27:40.584 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Plotfiles.html 2023-11-08T12:27:40.584 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2023-11-08T12:27:40.584 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2023-11-08T12:27:40.585 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Data-verbose.html 2023-11-08T12:27:40.585 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Data.html 2023-11-08T12:27:40.585 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Exceptions.html 2023-11-08T12:27:40.585 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/RunningAsAService.html 2023-11-08T12:27:40.585 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2023-11-08T12:27:40.586 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Ganglia.jpg 2023-11-08T12:27:40.586 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/OperationalModes.html 2023-11-08T12:27:40.586 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Playback.html 2023-11-08T12:27:40.586 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Hello.html 2023-11-08T12:27:40.586 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Sockets.html 2023-11-08T12:27:40.587 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Network.html 2023-11-08T12:27:40.587 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Infiniband.html 2023-11-08T12:27:40.587 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Memory.html 2023-11-08T12:27:40.587 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2023-11-08T12:27:40.587 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/Disks.html 2023-11-08T12:27:40.587 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/docs/NfsInfo.html 2023-11-08T12:27:40.588 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/service/ 2023-11-08T12:27:40.588 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/service/collectl.service 2023-11-08T12:27:40.588 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/initd/ 2023-11-08T12:27:40.588 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/initd/collectl-suse 2023-11-08T12:27:40.588 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/initd/collectl-debian 2023-11-08T12:27:40.589 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/initd/collectl-generic 2023-11-08T12:27:40.589 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/initd/collectl 2023-11-08T12:27:40.589 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/misc.ph 2023-11-08T12:27:40.589 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/gexpr.ph 2023-11-08T12:27:40.589 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/statsd.ph 2023-11-08T12:27:40.589 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/envrules.std 2023-11-08T12:27:40.590 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/hello.ph 2023-11-08T12:27:40.591 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/vmsum.ph 2023-11-08T12:27:40.591 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/proctree.ph 2023-11-08T12:27:40.591 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/README-WINDOWS 2023-11-08T12:27:40.591 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/formatit.ph 2023-11-08T12:27:40.593 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/colmux 2023-11-08T12:27:40.593 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/INSTALL 2023-11-08T12:27:40.594 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/graphite.ph 2023-11-08T12:27:40.594 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/RELEASE-collectl 2023-11-08T12:27:40.594 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/vmstat.ph 2023-11-08T12:27:40.594 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/README 2023-11-08T12:27:40.595 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/man1/ 2023-11-08T12:27:40.595 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/man1/collectl.1 2023-11-08T12:27:40.595 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/man1/colmux.1 2023-11-08T12:27:40.595 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/lexpr.ph 2023-11-08T12:27:40.595 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/client.pl 2023-11-08T12:27:40.595 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/collectl 2023-11-08T12:27:40.596 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/COPYING 2023-11-08T12:27:40.596 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/vnet.ph 2023-11-08T12:27:40.596 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/GPL 2023-11-08T12:27:40.596 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/UNINSTALL 2023-11-08T12:27:40.596 INFO:teuthology.orchestra.run.smithi039.stdout:collectl-4.3.1/ARTISTIC 2023-11-08T12:27:40.644 INFO:teuthology.orchestra.run.smithi039.stderr:./INSTALL: line 90: chkconfig: command not found 2023-11-08T12:27:40.646 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2023-11-08T12:27:41.247 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:03:26 ago on Wed 08 Nov 2023 12:24:15 PM UTC. 2023-11-08T12:27:41.347 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2023-11-08T12:27:41.349 INFO:teuthology.orchestra.run.smithi039.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2023-11-08T12:27:41.407 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-11-08T12:27:41.408 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:27:41.408 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-11-08T12:27:41.408 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:27:41.408 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-11-08T12:27:41.409 INFO:teuthology.orchestra.run.smithi039.stdout: librbd-devel x86_64 2:18.0.0-7227.g25e18238.el9 ceph 23 k 2023-11-08T12:27:41.409 INFO:teuthology.orchestra.run.smithi039.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2023-11-08T12:27:41.409 INFO:teuthology.orchestra.run.smithi039.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2023-11-08T12:27:41.409 INFO:teuthology.orchestra.run.smithi039.stdout: perf x86_64 5.14.0-382.el9 CentOS-AppStream 7.4 M 2023-11-08T12:27:41.409 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-11-08T12:27:41.410 INFO:teuthology.orchestra.run.smithi039.stdout: libradospp-devel x86_64 2:18.0.0-7227.g25e18238.el9 ceph 30 k 2023-11-08T12:27:41.410 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:41.410 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-11-08T12:27:41.410 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2023-11-08T12:27:41.410 INFO:teuthology.orchestra.run.smithi039.stdout:Install 5 Packages 2023-11-08T12:27:41.411 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:41.411 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 7.6 M 2023-11-08T12:27:41.412 INFO:teuthology.orchestra.run.smithi039.stdout:Installed size: 11 M 2023-11-08T12:27:41.412 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-11-08T12:27:41.953 INFO:teuthology.orchestra.run.smithi039.stdout:(1/5): libradospp-devel-18.0.0-7227.g25e18238.e 161 kB/s | 30 kB 00:00 2023-11-08T12:27:42.054 INFO:teuthology.orchestra.run.smithi039.stdout:(2/5): perf-5.14.0-382.el9.x86_64.rpm 26 MB/s | 7.4 MB 00:00 2023-11-08T12:27:42.095 INFO:teuthology.orchestra.run.smithi039.stdout:(3/5): librbd-devel-18.0.0-7227.g25e18238.el9.x 70 kB/s | 23 kB 00:00 2023-11-08T12:27:42.187 INFO:teuthology.orchestra.run.smithi039.stdout:(4/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 98 kB/s | 13 kB 00:00 2023-11-08T12:27:42.263 INFO:teuthology.orchestra.run.smithi039.stdout:(5/5): pdsh-2.34-7.el9.x86_64.rpm 386 kB/s | 118 kB 00:00 2023-11-08T12:27:42.264 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-11-08T12:27:42.265 INFO:teuthology.orchestra.run.smithi039.stdout:Total 8.9 MB/s | 7.6 MB 00:00 2023-11-08T12:27:42.265 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-11-08T12:27:42.287 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-11-08T12:27:42.287 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-11-08T12:27:42.448 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-11-08T12:27:42.449 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-11-08T12:27:42.768 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-11-08T12:27:42.839 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2023-11-08T12:27:42.879 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2023-11-08T12:27:42.910 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libradospp-devel-2:18.0.0-7227.g25e18238.el9.x86_64 3/5 2023-11-08T12:27:43.255 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librbd-devel-2:18.0.0-7227.g25e18238.el9.x86_64 4/5 2023-11-08T12:27:43.392 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : perf-5.14.0-382.el9.x86_64 5/5 2023-11-08T12:27:44.442 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: perf-5.14.0-382.el9.x86_64 5/5 2023-11-08T12:27:44.443 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libradospp-devel-2:18.0.0-7227.g25e18238.el9.x86_64 1/5 2023-11-08T12:27:44.443 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd-devel-2:18.0.0-7227.g25e18238.el9.x86_64 2/5 2023-11-08T12:27:44.443 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : perf-5.14.0-382.el9.x86_64 3/5 2023-11-08T12:27:44.443 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2023-11-08T12:27:44.815 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2023-11-08T12:27:44.815 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:44.816 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-11-08T12:27:44.816 INFO:teuthology.orchestra.run.smithi039.stdout: libradospp-devel-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:27:44.816 INFO:teuthology.orchestra.run.smithi039.stdout: librbd-devel-2:18.0.0-7227.g25e18238.el9.x86_64 2023-11-08T12:27:44.816 INFO:teuthology.orchestra.run.smithi039.stdout: pdsh-2.34-7.el9.x86_64 2023-11-08T12:27:44.816 INFO:teuthology.orchestra.run.smithi039.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2023-11-08T12:27:44.817 INFO:teuthology.orchestra.run.smithi039.stdout: perf-5.14.0-382.el9.x86_64 2023-11-08T12:27:44.817 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-11-08T12:27:44.817 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-11-08T12:27:45.214 INFO:tasks.cbt:benchmark: radosbench 2023-11-08T12:27:45.215 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2023-11-08T12:27:45.246 DEBUG:teuthology.orchestra.run.smithi039:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-11-08T12:27:45.654 INFO:teuthology.orchestra.run.smithi039.stderr:12:27:45 - DEBUG - cbt - Settings.cluster: 2023-11-08T12:27:45.655 INFO:teuthology.orchestra.run.smithi039.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2023-11-08T12:27:45.655 INFO:teuthology.orchestra.run.smithi039.stderr: 'clients': ['ubuntu@smithi039.front.sepia.ceph.com'], 2023-11-08T12:27:45.655 INFO:teuthology.orchestra.run.smithi039.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2023-11-08T12:27:45.655 INFO:teuthology.orchestra.run.smithi039.stderr: 'head': 'ubuntu@smithi039.front.sepia.ceph.com', 2023-11-08T12:27:45.655 INFO:teuthology.orchestra.run.smithi039.stderr: 'is_teuthology': True, 2023-11-08T12:27:45.655 INFO:teuthology.orchestra.run.smithi039.stderr: 'iterations': 1, 2023-11-08T12:27:45.656 INFO:teuthology.orchestra.run.smithi039.stderr: 'mons': ['ubuntu@smithi039.front.sepia.ceph.com'], 2023-11-08T12:27:45.656 INFO:teuthology.orchestra.run.smithi039.stderr: 'osds': ['ubuntu@smithi039.front.sepia.ceph.com'], 2023-11-08T12:27:45.656 INFO:teuthology.orchestra.run.smithi039.stderr: 'osds_per_node': 3, 2023-11-08T12:27:45.656 INFO:teuthology.orchestra.run.smithi039.stderr: 'pid_dir': '/var/run/ceph', 2023-11-08T12:27:45.656 INFO:teuthology.orchestra.run.smithi039.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2023-11-08T12:27:45.656 INFO:teuthology.orchestra.run.smithi039.stderr: 'pgp_size': 256, 2023-11-08T12:27:45.657 INFO:teuthology.orchestra.run.smithi039.stderr: 'replication': 'replicated'}}, 2023-11-08T12:27:45.657 INFO:teuthology.orchestra.run.smithi039.stderr: 'rebuild_every_test': False, 2023-11-08T12:27:45.657 INFO:teuthology.orchestra.run.smithi039.stderr: 'rgws': {'ubuntu@smithi039.front.sepia.ceph.com': None}, 2023-11-08T12:27:45.657 INFO:teuthology.orchestra.run.smithi039.stderr: 'tmp_dir': '/tmp/cbt', 2023-11-08T12:27:45.657 INFO:teuthology.orchestra.run.smithi039.stderr: 'use_existing': True, 2023-11-08T12:27:45.657 INFO:teuthology.orchestra.run.smithi039.stderr: 'user': 'ubuntu'} 2023-11-08T12:27:45.676 INFO:teuthology.orchestra.run.smithi039.stderr:12:27:45 - INFO - cbt - Pausing for 60s for idle monitoring. 2023-11-08T12:27:45.677 INFO:teuthology.orchestra.run.smithi039.stderr:12:27:45 - DEBUG - cbt - Nodes : ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:27:45.678 INFO:teuthology.orchestra.run.smithi039.stderr:12:27:45 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:27:45.678 INFO:teuthology.orchestra.run.smithi039.stderr:12:27:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl 2023-11-08T12:27:45.681 INFO:teuthology.orchestra.run.smithi039.stderr:12:27:45 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:27:45.681 INFO:teuthology.orchestra.run.smithi039.stderr:12:27:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-11-08T12:27:45.683 INFO:teuthology.orchestra.run.smithi039.stderr:12:27:45 - DEBUG - cbt - Nodes : ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:27:45.683 INFO:teuthology.orchestra.run.smithi039.stderr:12:27:45 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:27:45.683 INFO:teuthology.orchestra.run.smithi039.stderr:12:27:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf 2023-11-08T12:27:45.685 INFO:teuthology.orchestra.run.smithi039.stderr:12:27:45 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:27:45.685 INFO:teuthology.orchestra.run.smithi039.stderr:12:27:45 - DEBUG - cbt - PerfMonitoring: in local_node 2023-11-08T12:27:45.686 INFO:teuthology.orchestra.run.smithi039.stderr:12:27:45 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-11-08T12:28:45.694 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:45.694 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-11-08T12:28:45.734 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:45.734 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-11-08T12:28:45.772 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:45.772 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2023-11-08T12:28:45.774 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:45.774 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2023-11-08T12:28:45.775 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - Nodes : ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:28:45.776 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:45.776 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/* 2023-11-08T12:28:45.796 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:45.797 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/$(basename ${i}).smithi039.front.sepia.ceph.com ; done 2023-11-08T12:28:45.801 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - Nodes : ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:28:45.801 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:45.801 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo killall -9 rados 2023-11-08T12:28:45.856 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - WARNING - cbt - sudo killall -9 rados 2023-11-08T12:28:45.857 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - WARNING - cbt - error 1 seen, continuing anyway... 2023-11-08T12:28:45.877 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb. 2023-11-08T12:28:45.877 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004 2023-11-08T12:28:45.878 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - Nodes : ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:28:45.878 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:45.878 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004 2023-11-08T12:28:45.897 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - Nodes : ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:28:45.898 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:45.898 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl 2023-11-08T12:28:45.900 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:45.900 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-11-08T12:28:45.901 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - Nodes : ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:28:45.902 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:45.902 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf 2023-11-08T12:28:45.905 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:45.905 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - PerfMonitoring: in local_node 2023-11-08T12:28:45.905 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-11-08T12:28:45.906 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - Nodes : ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:28:45.906 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:45.906 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:45 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2023-11-08T12:28:46.210 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:46 - DEBUG - cbt - Nodes : ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:28:46.211 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:46 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:46.211 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:46 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2023-11-08T12:28:46.670 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:46 - DEBUG - cbt - Nodes : ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:28:46.671 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:46 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:46.671 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:46 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2023-11-08T12:28:47.677 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:47 - INFO - cbt - Checking Health after pool creation. 2023-11-08T12:28:52.683 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:52 - DEBUG - cbt - Nodes : ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:28:52.684 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:52 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:52.684 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:52 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-11-08T12:28:53.003 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:53 - INFO - cbt - Final Pool Health Check. 2023-11-08T12:28:58.007 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:58 - DEBUG - cbt - Nodes : ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:28:58.007 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:58 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:58.008 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-11-08T12:28:58.323 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:58 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:58.324 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-11-08T12:28:58.340 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:58 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:58.340 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-11-08T12:28:58.372 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:58 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:58.372 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2023-11-08T12:28:58.373 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:58 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:58.374 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2023-11-08T12:28:58.375 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:58 - DEBUG - cbt - Nodes : ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:28:58.375 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:58 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:58.375 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2023-11-08T12:28:58.473 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:58 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:58.474 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2023-11-08T12:28:59.195 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:59 - DEBUG - cbt - Nodes : ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:28:59.195 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:59 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:59.195 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2023-11-08T12:28:59.488 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:59 - DEBUG - cbt - Nodes : ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:28:59.489 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:59 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:59.489 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:59 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write 2023-11-08T12:28:59.537 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:59 - DEBUG - cbt - Nodes : ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:28:59.537 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:59 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:28:59.537 INFO:teuthology.orchestra.run.smithi039.stderr:12:28:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/ceph_settings.out 2023-11-08T12:29:00.336 INFO:teuthology.orchestra.run.smithi039.stderr:12:29:00 - DEBUG - cbt - Nodes : ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:29:00.336 INFO:teuthology.orchestra.run.smithi039.stderr:12:29:00 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:29:00.337 INFO:teuthology.orchestra.run.smithi039.stderr:12:29:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/collectl 2023-11-08T12:29:00.339 INFO:teuthology.orchestra.run.smithi039.stderr:12:29:00 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:29:00.339 INFO:teuthology.orchestra.run.smithi039.stderr:12:29:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-11-08T12:29:00.341 INFO:teuthology.orchestra.run.smithi039.stderr:12:29:00 - DEBUG - cbt - Nodes : ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:29:00.341 INFO:teuthology.orchestra.run.smithi039.stderr:12:29:00 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:29:00.341 INFO:teuthology.orchestra.run.smithi039.stderr:12:29:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf 2023-11-08T12:29:00.344 INFO:teuthology.orchestra.run.smithi039.stderr:12:29:00 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:29:00.344 INFO:teuthology.orchestra.run.smithi039.stderr:12:29:00 - DEBUG - cbt - PerfMonitoring: in local_node 2023-11-08T12:29:00.345 INFO:teuthology.orchestra.run.smithi039.stderr:12:29:00 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-11-08T12:29:00.345 INFO:teuthology.orchestra.run.smithi039.stderr:12:29:00 - INFO - cbt - Running radosbench write test. 2023-11-08T12:29:00.345 INFO:teuthology.orchestra.run.smithi039.stderr:12:29:00 - DEBUG - cbt - Nodes : ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:29:00.345 INFO:teuthology.orchestra.run.smithi039.stderr:12:29:00 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:29:00.346 INFO:teuthology.orchestra.run.smithi039.stderr:12:29:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/output.0 2023-11-08T12:29:00.346 INFO:teuthology.orchestra.run.smithi039.stderr:12:29:00 - DEBUG - cbt - Nodes : ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:29:00.347 INFO:teuthology.orchestra.run.smithi039.stderr:12:29:00 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:29:00.347 INFO:teuthology.orchestra.run.smithi039.stderr:12:29:00 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/output.1 2023-11-08T12:30:01.448 INFO:teuthology.orchestra.run.smithi039.stderr:12:30:01 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:30:01.449 INFO:teuthology.orchestra.run.smithi039.stderr:12:30:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-11-08T12:30:01.483 INFO:teuthology.orchestra.run.smithi039.stderr:12:30:01 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:30:01.483 INFO:teuthology.orchestra.run.smithi039.stderr:12:30:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-11-08T12:30:01.517 INFO:teuthology.orchestra.run.smithi039.stderr:12:30:01 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:30:01.517 INFO:teuthology.orchestra.run.smithi039.stderr:12:30:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf/perf.data 2023-11-08T12:30:01.518 INFO:teuthology.orchestra.run.smithi039.stderr:12:30:01 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:30:01.518 INFO:teuthology.orchestra.run.smithi039.stderr:12:30:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf/perf_stat.* 2023-11-08T12:30:01.520 INFO:teuthology.orchestra.run.smithi039.stderr:12:30:01 - DEBUG - cbt - Nodes : ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:30:01.520 INFO:teuthology.orchestra.run.smithi039.stderr:12:30:01 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:30:01.520 INFO:teuthology.orchestra.run.smithi039.stderr:12:30:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/historic_ops.out 2023-11-08T12:30:01.834 INFO:teuthology.orchestra.run.smithi039.stderr:12:30:01 - DEBUG - cbt - Nodes : ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:30:01.846 INFO:teuthology.orchestra.run.smithi039.stderr:12:30:01 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:30:01.846 INFO:teuthology.orchestra.run.smithi039.stderr:12:30:01 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/* 2023-11-08T12:30:01.865 INFO:teuthology.orchestra.run.smithi039.stderr:12:30:01 - DEBUG - cbt - get_fqdn_local()=smithi039.front.sepia.ceph.com 2023-11-08T12:30:01.866 INFO:teuthology.orchestra.run.smithi039.stderr:12:30:01 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/$(basename ${i}).smithi039.front.sepia.ceph.com ; done 2023-11-08T12:30:01.888 INFO:teuthology.orchestra.run.smithi039.stderr:12:30:01 - INFO - cbt - Convert results to json format. 2023-11-08T12:30:01.888 INFO:teuthology.orchestra.run.smithi039.stderr:12:30:01 - DEBUG - cbt - Nodes : ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:30:01.904 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2023-11-08T12:30:01.914 DEBUG:teuthology.orchestra.run.smithi039:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2023-11-08T12:30:02.034 INFO:tasks.cbt_performance:Collecting CBT performance data config 2023-11-08T12:30:02.034 INFO:tasks.cbt_performance:reading cbt preferences 2023-11-08T12:30:02.035 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest -name 'json_output.*' 2023-11-08T12:30:02.080 INFO:teuthology.orchestra.run.smithi039.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.0.smithi039.front.sepia.ceph.com 2023-11-08T12:30:02.080 INFO:teuthology.orchestra.run.smithi039.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.1.smithi039.front.sepia.ceph.com 2023-11-08T12:30:02.081 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-11-08T12:30:02.082 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.0.smithi039.front.sepia.ceph.com of=/dev/stdout 2023-11-08T12:30:02.102 INFO:tasks.cbt_performance:reading total cpu cycles 2023-11-08T12:30:02.103 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write -name 'perf_stat.*' 2023-11-08T12:30:02.158 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-11-08T12:30:02.159 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-11-08T12:30:02.159 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-11-08T12:30:02.160 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/json_output.1.smithi039.front.sepia.ceph.com of=/dev/stdout 2023-11-08T12:30:02.213 INFO:tasks.cbt_performance:reading total cpu cycles 2023-11-08T12:30:02.214 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write -name 'perf_stat.*' 2023-11-08T12:30:02.269 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-11-08T12:30:02.269 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-11-08T12:30:02.278 INFO:tasks.cbt_performance:Data inserted successfully. 2023-11-08T12:30:02.284 INFO:tasks.cbt_performance:Data inserted successfully. 2023-11-08T12:30:02.285 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-08T12:30:02.297 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-11-08T12:30:02.421 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:30:02.415+0000 7fbc33ade640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:30:02.423 INFO:teuthology.orchestra.run.smithi039.stderr:2023-11-08T12:30:02.423+0000 7fbc33ade640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-08T12:30:02.626 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-08T12:30:02.627 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-08T12:30:02.627 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-11-08T12:30:02.628 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T12:30:05.195 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-11-08T12:30:08.729 INFO:tasks.ceph.osd.0:Stopped 2023-11-08T12:30:08.730 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-11-08T12:30:08.731 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T12:30:10.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-11-08T12:30:10.498 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-11-08T12:30:14.832 INFO:tasks.ceph.osd.1:Stopped 2023-11-08T12:30:14.833 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-11-08T12:30:14.833 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T12:30:15.700 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-11-08T12:30:15.701 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-11-08T12:30:20.903 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~5s 2023-11-08T12:30:20.935 INFO:tasks.ceph.osd.2:Stopped 2023-11-08T12:30:20.935 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-08T12:30:20.936 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-11-08T12:30:20.936 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T12:30:20.968 INFO:tasks.ceph.mgr.x:Stopped 2023-11-08T12:30:20.969 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-08T12:30:20.970 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-11-08T12:30:20.970 INFO:teuthology.orchestra.run:waiting for 300 2023-11-08T12:30:21.022 INFO:tasks.ceph.mon.a:Stopped 2023-11-08T12:30:21.022 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-08T12:30:21.023 DEBUG:teuthology.orchestra.run.smithi039:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-11-08T12:30:21.107 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:30:21.108 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-11-08T12:30:21.601 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:30:21.602 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-11-08T12:30:21.694 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:30:21.694 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-11-08T12:30:21.762 INFO:tasks.ceph:Archiving mon data... 2023-11-08T12:30:21.763 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2023-11-08_11:46:17-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7451942/data/mon.a.tgz 2023-11-08T12:30:21.764 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2023-11-08T12:30:21.783 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.dPmdFxuI34 2023-11-08T12:30:21.783 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.dPmdFxuI34 2023-11-08T12:30:21.921 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.dPmdFxuI34 2023-11-08T12:30:22.025 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.dPmdFxuI34 is 176KB 2023-11-08T12:30:22.076 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.dPmdFxuI34 2023-11-08T12:30:22.092 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-08T12:30:22.092 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-08T12:30:22.163 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-11-08T12:30:22.163 DEBUG:teuthology.orchestra.run.smithi039:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-08T12:30:22.258 INFO:teuthology.orchestra.run.smithi039.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-08T12:30:22.263 INFO:tasks.ceph:Archiving crash dumps... 2023-11-08T12:30:22.265 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2023-11-08_11:46:17-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7451942/remote/smithi039/crash 2023-11-08T12:30:22.266 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-08T12:30:22.296 INFO:tasks.ceph:Compressing logs... 2023-11-08T12:30:22.296 DEBUG:teuthology.orchestra.run.smithi039:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-08T12:30:22.372 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-11-08T12:30:22.373 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80941.log 2023-11-08T12:30:22.373 INFO:teuthology.orchestra.run.smithi039.stderr:gzip/var/log/ceph/ceph-mon.a.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.80831.log 2023-11-08T12:30:22.374 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.80941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83416.log 2023-11-08T12:30:22.375 INFO:teuthology.orchestra.run.smithi039.stderr:gzip/var/log/ceph/ceph-client.admin.80831.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.81939.log 2023-11-08T12:30:22.375 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.83416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81209.log 2023-11-08T12:30:22.375 INFO:teuthology.orchestra.run.smithi039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83416.log.gz 2023-11-08T12:30:22.375 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81179.log 2023-11-08T12:30:22.375 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82153.log 2023-11-08T12:30:22.376 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83029.log 2023-11-08T12:30:22.376 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81672.log 2023-11-08T12:30:22.376 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83064.log 2023-11-08T12:30:22.376 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-08T12:30:22.380 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82065.log 2023-11-08T12:30:22.381 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82067.log 2023-11-08T12:30:22.381 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.79380.log 2023-11-08T12:30:22.381 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-11-08T12:30:22.381 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80981.log 2023-11-08T12:30:22.382 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81940.log 2023-11-08T12:30:22.382 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81775.log 2023-11-08T12:30:22.382 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82195.log 2023-11-08T12:30:22.382 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81859.log 2023-11-08T12:30:22.382 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81714.log 2023-11-08T12:30:22.383 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81941.log 2023-11-08T12:30:22.383 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81616.log 2023-11-08T12:30:22.383 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80743.log 2023-11-08T12:30:22.383 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82237.log 2023-11-08T12:30:22.383 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-11-08T12:30:22.384 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82964.log 2023-11-08T12:30:22.384 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83179.log 2023-11-08T12:30:22.384 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83178.log 2023-11-08T12:30:22.384 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82066.log 2023-11-08T12:30:22.384 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-11-08T12:30:22.385 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81817.log 2023-11-08T12:30:22.385 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-08T12:30:22.385 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-11-08T12:30:22.385 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81021.log 2023-11-08T12:30:22.385 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83114.log 2023-11-08T12:30:22.385 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83090.log 2023-11-08T12:30:22.394 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.82237.log: /var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.81817.log: /var/log/ceph/ceph-client.admin.81616.log: /var/log/ceph/ceph-client.admin.80743.log: /var/log/ceph/ceph-client.admin.82964.log: /var/log/ceph/ceph-client.admin.81021.log: /var/log/ceph/ceph-client.admin.81209.log: /var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.83090.log: /var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.83179.log: /var/log/ceph/ceph-client.admin.83114.log: /var/log/ceph/ceph-client.admin.83178.log: /var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph.log: /var/log/ceph/ceph-osd.2.log: 50.4% 66.7% 50.4% -- replaced with /var/log/ceph/ceph-client.admin.80831.log.gz -- replaced with /var/log/ceph/ceph-client.admin.83179.log.gz 2023-11-08T12:30:22.395 INFO:teuthology.orchestra.run.smithi039.stderr: 2023-11-08T12:30:22.395 INFO:teuthology.orchestra.run.smithi039.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.83178.log.gz 2023-11-08T12:30:22.395 INFO:teuthology.orchestra.run.smithi039.stderr: -- replaced with /var/log/ceph/ceph-client.admin.80941.log.gz/var/log/ceph/ceph-client.admin.81939.log: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-11-08T12:30:22.395 INFO:teuthology.orchestra.run.smithi039.stderr: 2023-11-08T12:30:22.407 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.83064.log: /var/log/ceph/ceph-client.admin.81179.log: /var/log/ceph/ceph-client.admin.82067.log: /var/log/ceph/ceph-client.admin.83029.log: /var/log/ceph/ceph-client.admin.82153.log: /var/log/ceph/ceph-client.admin.81672.log: /var/log/ceph/ceph-client.admin.82065.log: /var/log/ceph/ceph-client.admin.80981.log: /var/log/ceph/ceph-client.admin.82066.log: /var/log/ceph/ceph-client.admin.81775.log: /var/log/ceph/ceph.tmp-client.admin.79380.log: /var/log/ceph/ceph-client.admin.81714.log: /var/log/ceph/ceph-client.admin.81940.log: /var/log/ceph/ceph-client.admin.81859.log: /var/log/ceph/ceph-client.admin.82195.log: /var/log/ceph/ceph-client.admin.81941.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.82964.log.gz 2023-11-08T12:30:22.407 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83029.log.gz 2023-11-08T12:30:22.408 INFO:teuthology.orchestra.run.smithi039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83064.log.gz 2023-11-08T12:30:22.415 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81179.log.gz 2023-11-08T12:30:22.416 INFO:teuthology.orchestra.run.smithi039.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.81209.log.gz 2023-11-08T12:30:22.420 INFO:teuthology.orchestra.run.smithi039.stderr: 87.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-08T12:30:22.423 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80981.log.gz 2023-11-08T12:30:22.424 INFO:teuthology.orchestra.run.smithi039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81021.log.gz 2023-11-08T12:30:22.447 INFO:teuthology.orchestra.run.smithi039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81616.log.gz 2023-11-08T12:30:22.447 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81672.log.gz 2023-11-08T12:30:22.448 INFO:teuthology.orchestra.run.smithi039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81714.log.gz 2023-11-08T12:30:22.448 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81775.log.gz 2023-11-08T12:30:22.448 INFO:teuthology.orchestra.run.smithi039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81817.log.gz 2023-11-08T12:30:22.448 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81859.log.gz 2023-11-08T12:30:22.448 INFO:teuthology.orchestra.run.smithi039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81941.log.gz 2023-11-08T12:30:22.449 INFO:teuthology.orchestra.run.smithi039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81940.log.gz 2023-11-08T12:30:22.449 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81939.log.gz 2023-11-08T12:30:22.449 INFO:teuthology.orchestra.run.smithi039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82067.log.gz 2023-11-08T12:30:22.449 INFO:teuthology.orchestra.run.smithi039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82065.log.gz 2023-11-08T12:30:22.449 INFO:teuthology.orchestra.run.smithi039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82066.log.gz 2023-11-08T12:30:22.449 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82153.log.gz 2023-11-08T12:30:22.450 INFO:teuthology.orchestra.run.smithi039.stderr: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.82195.log.gz 2023-11-08T12:30:22.450 INFO:teuthology.orchestra.run.smithi039.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82237.log.gz 2023-11-08T12:30:22.459 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80743.log.gz 2023-11-08T12:30:22.459 INFO:teuthology.orchestra.run.smithi039.stderr: 85.4% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-08T12:30:22.463 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83090.log.gz 2023-11-08T12:30:22.463 INFO:teuthology.orchestra.run.smithi039.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83114.log.gz 2023-11-08T12:30:22.489 INFO:teuthology.orchestra.run.smithi039.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-11-08T12:30:22.566 INFO:teuthology.orchestra.run.smithi039.stderr: 68.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.79380.log.gz 2023-11-08T12:30:23.763 INFO:teuthology.orchestra.run.smithi039.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-11-08T12:30:23.774 INFO:teuthology.orchestra.run.smithi039.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-11-08T12:30:23.828 INFO:teuthology.orchestra.run.smithi039.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-11-08T12:30:23.829 INFO:teuthology.orchestra.run.smithi039.stderr: 2023-11-08T12:30:23.830 INFO:teuthology.orchestra.run.smithi039.stderr:real 0m1.478s 2023-11-08T12:30:23.830 INFO:teuthology.orchestra.run.smithi039.stderr:user 0m2.846s 2023-11-08T12:30:23.830 INFO:teuthology.orchestra.run.smithi039.stderr:sys 0m0.129s 2023-11-08T12:30:23.830 INFO:tasks.ceph:Archiving logs... 2023-11-08T12:30:23.831 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/log/ceph to /home/teuthworker/archive/matan-2023-11-08_11:46:17-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7451942/remote/smithi039/log 2023-11-08T12:30:23.832 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-08T12:30:24.361 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-08T12:30:24.371 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-08T12:30:24.371 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-08T12:30:24.429 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-08T12:30:24.439 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-08T12:30:24.439 DEBUG:teuthology.orchestra.run.smithi039:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-08T12:30:24.458 INFO:teuthology.orchestra.run.smithi039.stderr:bash: line 1: ntpq: command not found 2023-11-08T12:30:24.503 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-08T12:30:24.504 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2023-11-08T12:30:24.504 INFO:teuthology.orchestra.run.smithi039.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 6 +935us[ +935us] +/- 82ms 2023-11-08T12:30:24.504 INFO:teuthology.orchestra.run.smithi039.stdout:^* hv02.front.sepia.ceph.com 3 6 377 5 -377us[ -377us] +/- 60ms 2023-11-08T12:30:24.505 INFO:teuthology.orchestra.run.smithi039.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 -385us[ -385us] +/- 80ms 2023-11-08T12:30:24.505 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-08T12:30:24.518 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-08T12:30:24.527 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-08T12:30:24.528 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-08T12:30:24.551 DEBUG:teuthology.orchestra.run.smithi039:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-08T12:30:24.725 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-08T12:30:24.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T12:30:24.761 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-08T12:30:24.771 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-08T12:30:24.799 INFO:teuthology.task.internal:Duration was 744.320457 seconds 2023-11-08T12:30:24.800 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-08T12:30:24.834 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-08T12:30:24.835 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-08T12:30:24.977 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-08T12:30:25.275 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-08T12:30:25.276 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi039.front.sepia.ceph.com 2023-11-08T12:30:25.278 DEBUG:teuthology.orchestra.run.smithi039:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-08T12:30:25.299 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-08T12:30:25.299 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-08T12:30:25.372 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-08T12:30:25.373 DEBUG:teuthology.orchestra.run.smithi039:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-08T12:30:25.520 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-08T12:30:25.531 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-08T12:30:25.531 DEBUG:teuthology.orchestra.run.smithi039:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-08T12:30:25.567 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-08T12:30:25.580 DEBUG:teuthology.orchestra.run.smithi039:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-08T12:30:25.646 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = core 2023-11-08T12:30:25.676 DEBUG:teuthology.orchestra.run.smithi039:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-08T12:30:25.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-08T12:30:25.731 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-08T12:30:25.741 INFO:teuthology.task.internal:Transferring archived files... 2023-11-08T12:30:25.742 DEBUG:teuthology.misc:Transferring archived files from smithi039:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2023-11-08_11:46:17-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7451942/remote/smithi039 2023-11-08T12:30:25.743 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-08T12:30:25.835 INFO:teuthology.task.internal:Removing archive directory... 2023-11-08T12:30:25.835 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-08T12:30:25.885 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-08T12:30:25.898 INFO:teuthology.task.internal:Not uploading archives. 2023-11-08T12:30:25.899 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-08T12:30:25.985 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-08T12:30:25.985 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-08T12:30:25.986 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-11-08T12:30:26.000 INFO:teuthology.orchestra.run.smithi039.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 8 12:30 /home/ubuntu/cephtest 2023-11-08T12:30:26.002 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-08T12:30:26.016 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-08T12:30:26.087 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/Rfy7LkB4k/rados-bench?orgId=1&var-branch_name=wip-matanb-crimson-do_osd_ops_execute-v3&var-job_id_selected=7451942 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} duration: 744.3204569816589 flavor: crimson owner: scheduled_matan@teuthology success: true 2023-11-08T12:30:26.087 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-08T12:30:26.195 INFO:teuthology.run:pass