2023-10-31T12:15:05.039 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-10-31T12:15:05.040 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_matan@teuthology --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_seq_read} --name matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi --archive /home/teuthworker/archive/matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7441974 -- /home/teuthworker/archive/matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7441974/orig.config.yaml 2023-10-31T12:15:05.084 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-31T12:15:05.162 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7441974 branch: wip-wip-matanb-crimson-do_osd_ops_execute-v3 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_seq_read} email: null first_in_suite: false job_id: '7441974' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-wip-matanb-crimson-do_osd_ops_execute-v3 ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: e83bf4821bfb604694b03a67d7390bfb38655506 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: e83bf4821bfb604694b03a67d7390bfb38655506 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: 21548fe806cf259deac1421530d5ce720be17997 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1530 sha1: e83bf4821bfb604694b03a67d7390bfb38655506 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_21548fe806cf259deac1421530d5ce720be17997/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 21548fe806cf259deac1421530d5ce720be17997 targets: smithi061.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLrJA40Hkx/RPJJwnSiZcODOwfqW/iC7Q/PzuwzQah+m/ONKi/hvZW56fzMf8XhGcoKh5u3N8Bc1cNqBLauDHrk= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4096 pool_monitoring_list: - collectl pool_profile: replicated run_monitoring_list: - collectl time: 60 write_only: false cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-10-31_11:40:33 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.770726 2023-10-31T12:15:05.162 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_21548fe806cf259deac1421530d5ce720be17997/qa; will attempt to use it 2023-10-31T12:15:05.163 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_21548fe806cf259deac1421530d5ce720be17997/qa/tasks 2023-10-31T12:15:05.163 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-31T12:15:05.165 INFO:teuthology.task.internal:Checking packages... 2023-10-31T12:15:05.189 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash 'e83bf4821bfb604694b03a67d7390bfb38655506' 2023-10-31T12:15:05.189 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-31T12:15:05.189 INFO:teuthology.packaging:ref: None 2023-10-31T12:15:05.190 INFO:teuthology.packaging:tag: None 2023-10-31T12:15:05.190 INFO:teuthology.packaging:branch: wip-wip-matanb-crimson-do_osd_ops_execute-v3 2023-10-31T12:15:05.190 INFO:teuthology.packaging:sha1: e83bf4821bfb604694b03a67d7390bfb38655506 2023-10-31T12:15:05.191 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-wip-matanb-crimson-do_osd_ops_execute-v3 2023-10-31T12:15:05.334 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6792.ge83bf482 2023-10-31T12:15:05.335 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-31T12:15:05.340 INFO:teuthology.task.internal:no buildpackages task found 2023-10-31T12:15:05.340 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-31T12:15:05.357 INFO:teuthology.task.internal:Saving configuration 2023-10-31T12:15:05.373 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-31T12:15:05.379 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-31T12:15:05.403 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi061.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7441974', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-31 12:10:00.948266', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLrJA40Hkx/RPJJwnSiZcODOwfqW/iC7Q/PzuwzQah+m/ONKi/hvZW56fzMf8XhGcoKh5u3N8Bc1cNqBLauDHrk='} 2023-10-31T12:15:05.403 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-31T12:15:05.409 INFO:teuthology.task.internal:roles: ubuntu@smithi061.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-31T12:15:05.409 INFO:teuthology.run_tasks:Running task console_log... 2023-10-31T12:15:05.447 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa7551e0940>, signals=[15]) 2023-10-31T12:15:05.450 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-31T12:15:05.455 INFO:teuthology.task.internal:Opening connections... 2023-10-31T12:15:05.456 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:15:05.457 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-31T12:15:05.558 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-31T12:15:05.564 DEBUG:teuthology.orchestra.run.smithi061:> uname -m 2023-10-31T12:15:05.627 INFO:teuthology.orchestra.run.smithi061.stdout:x86_64 2023-10-31T12:15:05.627 DEBUG:teuthology.orchestra.run.smithi061:> cat /etc/os-release 2023-10-31T12:15:05.683 INFO:teuthology.orchestra.run.smithi061.stdout:NAME="CentOS Stream" 2023-10-31T12:15:05.684 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION="9" 2023-10-31T12:15:05.684 INFO:teuthology.orchestra.run.smithi061.stdout:ID="centos" 2023-10-31T12:15:05.684 INFO:teuthology.orchestra.run.smithi061.stdout:ID_LIKE="rhel fedora" 2023-10-31T12:15:05.684 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION_ID="9" 2023-10-31T12:15:05.684 INFO:teuthology.orchestra.run.smithi061.stdout:PLATFORM_ID="platform:el9" 2023-10-31T12:15:05.685 INFO:teuthology.orchestra.run.smithi061.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-31T12:15:05.685 INFO:teuthology.orchestra.run.smithi061.stdout:ANSI_COLOR="0;31" 2023-10-31T12:15:05.685 INFO:teuthology.orchestra.run.smithi061.stdout:LOGO="fedora-logo-icon" 2023-10-31T12:15:05.685 INFO:teuthology.orchestra.run.smithi061.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-31T12:15:05.685 INFO:teuthology.orchestra.run.smithi061.stdout:HOME_URL="https://centos.org/" 2023-10-31T12:15:05.686 INFO:teuthology.orchestra.run.smithi061.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-31T12:15:05.686 INFO:teuthology.orchestra.run.smithi061.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-31T12:15:05.686 INFO:teuthology.orchestra.run.smithi061.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-31T12:15:05.687 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server 2023-10-31T12:15:05.712 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-31T12:15:05.720 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-31T12:15:05.725 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-31T12:15:05.725 DEBUG:teuthology.orchestra.run.smithi061:> test '!' -e /home/ubuntu/cephtest 2023-10-31T12:15:05.741 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-31T12:15:05.747 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-31T12:15:05.747 DEBUG:teuthology.orchestra.run.smithi061:> test -z $(ls -A /var/lib/ceph) 2023-10-31T12:15:05.805 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-31T12:15:05.837 INFO:teuthology.run_tasks:Running task kernel... 2023-10-31T12:15:05.852 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-31T12:15:05.852 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-31T12:15:05.853 DEBUG:teuthology.orchestra.run.smithi061:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-31T12:15:05.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-31T12:15:05.869 DEBUG:teuthology.orchestra.run.smithi061:> uname -r 2023-10-31T12:15:05.926 INFO:teuthology.orchestra.run.smithi061.stdout:5.14.0-361.el9.x86_64 2023-10-31T12:15:05.926 INFO:teuthology.task.kernel:Running kernel on smithi061: 5.14.0-361.el9.x86_64 2023-10-31T12:15:05.927 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum install -y kernel 2023-10-31T12:15:07.888 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS-9 - AppStream 16 MB/s | 6.3 MB 00:00 2023-10-31T12:15:09.272 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS-9 - BaseOS 7.8 MB/s | 2.0 MB 00:00 2023-10-31T12:15:12.604 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 9 - BaseOS 2.6 MB/s | 7.9 MB 00:03 2023-10-31T12:15:16.255 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 9 - AppStream 6.9 MB/s | 18 MB 00:02 2023-10-31T12:15:22.030 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 5.8 MB 00:02 2023-10-31T12:15:23.406 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 9 - Extras packages 74 kB/s | 15 kB 00:00 2023-10-31T12:15:25.041 INFO:teuthology.orchestra.run.smithi061.stdout:Extra Packages for Enterprise Linux 13 MB/s | 19 MB 00:01 2023-10-31T12:15:29.699 INFO:teuthology.orchestra.run.smithi061.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2023-10-31T12:15:31.915 INFO:teuthology.orchestra.run.smithi061.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-10-31T12:15:31.915 INFO:teuthology.orchestra.run.smithi061.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-10-31T12:15:32.019 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:15:32.028 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:15:32.028 INFO:teuthology.orchestra.run.smithi061.stdout: Package Architecture Version Repository Size 2023-10-31T12:15:32.028 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:15:32.029 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-10-31T12:15:32.029 INFO:teuthology.orchestra.run.smithi061.stdout: kernel x86_64 5.14.0-378.el9 baseos 5.3 M 2023-10-31T12:15:32.029 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-10-31T12:15:32.029 INFO:teuthology.orchestra.run.smithi061.stdout: kernel-core x86_64 5.14.0-378.el9 baseos 20 M 2023-10-31T12:15:32.029 INFO:teuthology.orchestra.run.smithi061.stdout: kernel-modules x86_64 5.14.0-378.el9 baseos 38 M 2023-10-31T12:15:32.030 INFO:teuthology.orchestra.run.smithi061.stdout: kernel-modules-core x86_64 5.14.0-378.el9 baseos 32 M 2023-10-31T12:15:32.030 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:15:32.030 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-10-31T12:15:32.031 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:15:32.031 INFO:teuthology.orchestra.run.smithi061.stdout:Install 4 Packages 2023-10-31T12:15:32.031 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:15:32.031 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 95 M 2023-10-31T12:15:32.031 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 120 M 2023-10-31T12:15:32.032 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-10-31T12:15:32.659 INFO:teuthology.orchestra.run.smithi061.stdout:(1/4): kernel-5.14.0-378.el9.x86_64.rpm 9.1 MB/s | 5.3 MB 00:00 2023-10-31T12:15:33.992 INFO:teuthology.orchestra.run.smithi061.stdout:(2/4): kernel-core-5.14.0-378.el9.x86_64.rpm 10 MB/s | 20 MB 00:01 2023-10-31T12:15:34.376 INFO:teuthology.orchestra.run.smithi061.stdout:(3/4): kernel-modules-core-5.14.0-378.el9.x86_6 19 MB/s | 32 MB 00:01 2023-10-31T12:15:34.718 INFO:teuthology.orchestra.run.smithi061.stdout:(4/4): kernel-modules-5.14.0-378.el9.x86_64.rpm 14 MB/s | 38 MB 00:02 2023-10-31T12:15:34.718 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:15:34.719 INFO:teuthology.orchestra.run.smithi061.stdout:Total 35 MB/s | 95 MB 00:02 2023-10-31T12:15:35.596 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-10-31T12:15:36.022 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-10-31T12:15:36.023 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-10-31T12:15:37.368 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-10-31T12:15:37.369 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-10-31T12:15:40.120 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-10-31T12:15:40.783 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : kernel-modules-core-5.14.0-378.el9.x86_64 1/4 2023-10-31T12:15:40.901 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-31T12:15:43.805 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-31T12:15:43.995 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-31T12:15:48.943 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-31T12:15:49.055 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : kernel-5.14.0-378.el9.x86_64 4/4 2023-10-31T12:15:54.036 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-31T12:16:24.506 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 4/4 2023-10-31T12:16:24.551 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 4/4 2023-10-31T12:16:25.018 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: kernel-5.14.0-378.el9.x86_64 4/4 2023-10-31T12:16:25.018 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : kernel-5.14.0-378.el9.x86_64 1/4 2023-10-31T12:16:25.018 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-10-31T12:16:25.019 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-10-31T12:16:25.434 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-10-31T12:16:25.435 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:16:25.435 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-10-31T12:16:25.435 INFO:teuthology.orchestra.run.smithi061.stdout: kernel-5.14.0-378.el9.x86_64 2023-10-31T12:16:25.435 INFO:teuthology.orchestra.run.smithi061.stdout: kernel-core-5.14.0-378.el9.x86_64 2023-10-31T12:16:25.435 INFO:teuthology.orchestra.run.smithi061.stdout: kernel-modules-5.14.0-378.el9.x86_64 2023-10-31T12:16:25.436 INFO:teuthology.orchestra.run.smithi061.stdout: kernel-modules-core-5.14.0-378.el9.x86_64 2023-10-31T12:16:25.436 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:16:25.436 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:16:25.933 DEBUG:teuthology.orchestra.run.smithi061:> rpm -q kernel | sort -rV | head -n 1 2023-10-31T12:16:26.077 INFO:teuthology.orchestra.run.smithi061.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-31T12:16:26.078 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-31T12:16:26.078 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-378.el9.x86_64 2023-10-31T12:16:26.078 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-31T12:16:26.079 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-31T12:16:26.079 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-31T12:16:26.079 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi061.front.sepia.ceph.com, path=None, version=distro) 2023-10-31T12:16:26.079 DEBUG:teuthology.orchestra.run.smithi061:> rpm -q kernel | sort -rV | head -n 1 2023-10-31T12:16:26.202 INFO:teuthology.orchestra.run.smithi061.stdout:kernel-5.14.0-378.el9.x86_64 2023-10-31T12:16:26.202 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-10-31T12:16:26.203 DEBUG:teuthology.orchestra.run.smithi061:> sudo rpm -qi grub2-tools 2023-10-31T12:16:26.241 INFO:teuthology.orchestra.run.smithi061.stdout:Name : grub2-tools 2023-10-31T12:16:26.241 INFO:teuthology.orchestra.run.smithi061.stdout:Epoch : 1 2023-10-31T12:16:26.241 INFO:teuthology.orchestra.run.smithi061.stdout:Version : 2.06 2023-10-31T12:16:26.242 INFO:teuthology.orchestra.run.smithi061.stdout:Release : 68.el9 2023-10-31T12:16:26.242 INFO:teuthology.orchestra.run.smithi061.stdout:Architecture: x86_64 2023-10-31T12:16:26.242 INFO:teuthology.orchestra.run.smithi061.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-31T12:16:26.242 INFO:teuthology.orchestra.run.smithi061.stdout:Group : Unspecified 2023-10-31T12:16:26.243 INFO:teuthology.orchestra.run.smithi061.stdout:Size : 8264254 2023-10-31T12:16:26.243 INFO:teuthology.orchestra.run.smithi061.stdout:License : GPLv3+ 2023-10-31T12:16:26.243 INFO:teuthology.orchestra.run.smithi061.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-31T12:16:26.244 INFO:teuthology.orchestra.run.smithi061.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-31T12:16:26.244 INFO:teuthology.orchestra.run.smithi061.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-31T12:16:26.244 INFO:teuthology.orchestra.run.smithi061.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-31T12:16:26.244 INFO:teuthology.orchestra.run.smithi061.stdout:Packager : builder@centos.org 2023-10-31T12:16:26.244 INFO:teuthology.orchestra.run.smithi061.stdout:Vendor : CentOS 2023-10-31T12:16:26.245 INFO:teuthology.orchestra.run.smithi061.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-31T12:16:26.245 INFO:teuthology.orchestra.run.smithi061.stdout:Summary : Support tools for GRUB. 2023-10-31T12:16:26.245 INFO:teuthology.orchestra.run.smithi061.stdout:Description : 2023-10-31T12:16:26.245 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:16:26.245 INFO:teuthology.orchestra.run.smithi061.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-31T12:16:26.245 INFO:teuthology.orchestra.run.smithi061.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-31T12:16:26.246 INFO:teuthology.orchestra.run.smithi061.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-31T12:16:26.246 INFO:teuthology.orchestra.run.smithi061.stdout:hardware devices. 2023-10-31T12:16:26.246 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:16:26.246 INFO:teuthology.orchestra.run.smithi061.stdout:This subpackage provides tools for support of all platforms. 2023-10-31T12:16:26.247 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-31T12:16:26.248 INFO:teuthology.task.kernel:Updating grub on smithi061 to boot 5.14.0-378.el9.x86_64 2023-10-31T12:16:26.248 DEBUG:teuthology.orchestra.run.smithi061:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-31T12:16:26.900 INFO:teuthology.orchestra.run.smithi061.stderr:Generating grub configuration file ... 2023-10-31T12:16:28.255 INFO:teuthology.orchestra.run.smithi061.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-31T12:16:28.288 INFO:teuthology.orchestra.run.smithi061.stderr:done 2023-10-31T12:16:28.290 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2023-10-31T12:16:28.306 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.Txr0BOW36g 2023-10-31T12:16:28.306 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Txr0BOW36g 2023-10-31T12:16:28.377 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.Txr0BOW36g 2023-10-31T12:16:28.518 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.Txr0BOW36g is 6KB 2023-10-31T12:16:28.568 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.Txr0BOW36g 2023-10-31T12:16:28.582 DEBUG:teuthology.orchestra.run.smithi061:> sudo /bin/ls /boot/loader/entries || true 2023-10-31T12:16:28.658 INFO:teuthology.orchestra.run.smithi061.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-31T12:16:28.658 INFO:teuthology.orchestra.run.smithi061.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-31T12:16:28.658 INFO:teuthology.orchestra.run.smithi061.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-31T12:16:28.659 INFO:teuthology.orchestra.run.smithi061.stdout:15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64.conf 2023-10-31T12:16:28.659 DEBUG:teuthology.orchestra.run.smithi061:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64 2023-10-31T12:16:28.813 DEBUG:teuthology.orchestra.run.smithi061:> sudo shutdown -r now 2023-10-31T12:16:58.847 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-31T12:16:58.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi061.front.sepia.ceph.com' 2023-10-31T12:16:58.848 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-31T12:17:17.178 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.61 2023-10-31T12:17:26.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi061.front.sepia.ceph.com' 2023-10-31T12:17:26.189 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-31T12:17:29.242 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.61 2023-10-31T12:17:41.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi061.front.sepia.ceph.com' 2023-10-31T12:17:41.256 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-31T12:17:59.674 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.61 2023-10-31T12:18:14.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi061.front.sepia.ceph.com' 2023-10-31T12:18:14.683 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-31T12:18:15.124 DEBUG:teuthology.orchestra.run.smithi061:> true 2023-10-31T12:18:15.527 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi061.front.sepia.ceph.com' 2023-10-31T12:18:15.527 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-378.el9.x86_64"... 2023-10-31T12:18:15.528 DEBUG:teuthology.orchestra.run.smithi061:> uname -r 2023-10-31T12:18:15.542 INFO:teuthology.orchestra.run.smithi061.stdout:5.14.0-378.el9.x86_64 2023-10-31T12:18:15.543 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-378.el9.x86_64 vs 5.14.0-378.el9.x86_64 2023-10-31T12:18:15.543 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-31T12:18:15.543 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-31T12:18:16.545 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-31T12:18:16.545 DEBUG:teuthology.orchestra.run.smithi061:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-31T12:18:16.610 INFO:teuthology.orchestra.run.smithi061.stdout:ttyS1 2023-10-31T12:18:16.675 DEBUG:teuthology.parallel:result is None 2023-10-31T12:18:16.676 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-31T12:18:16.683 INFO:teuthology.task.internal:Creating test directory... 2023-10-31T12:18:16.683 DEBUG:teuthology.orchestra.run.smithi061:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-31T12:18:16.698 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-31T12:18:16.705 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-31T12:18:16.712 INFO:teuthology.task.internal:Creating archive directory... 2023-10-31T12:18:16.712 DEBUG:teuthology.orchestra.run.smithi061:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-31T12:18:16.838 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-31T12:18:16.845 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-31T12:18:16.845 DEBUG:teuthology.orchestra.run.smithi061:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-31T12:18:16.889 INFO:teuthology.orchestra.run.smithi061.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-31T12:18:16.905 INFO:teuthology.orchestra.run.smithi061.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-31T12:18:16.910 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-31T12:18:16.916 INFO:teuthology.task.internal:Configuring sudo... 2023-10-31T12:18:16.917 DEBUG:teuthology.orchestra.run.smithi061:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-31T12:18:16.950 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-31T12:18:16.959 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-31T12:18:16.960 DEBUG:teuthology.orchestra.run.smithi061:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-31T12:18:17.003 DEBUG:teuthology.orchestra.run.smithi061:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-31T12:18:17.082 DEBUG:teuthology.orchestra.run.smithi061:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-31T12:18:17.133 DEBUG:teuthology.orchestra.run.smithi061:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-31T12:18:17.213 DEBUG:teuthology.orchestra.run.smithi061:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-31T12:18:17.254 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:18:17.254 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-31T12:18:17.321 DEBUG:teuthology.orchestra.run.smithi061:> sudo service rsyslog restart 2023-10-31T12:18:17.409 INFO:teuthology.orchestra.run.smithi061.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-31T12:18:17.849 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-31T12:18:17.857 INFO:teuthology.task.internal:Starting timer... 2023-10-31T12:18:17.857 INFO:teuthology.run_tasks:Running task pcp... 2023-10-31T12:18:17.868 INFO:teuthology.run_tasks:Running task selinux... 2023-10-31T12:18:17.876 DEBUG:teuthology.orchestra.run.smithi061:> sudo service auditd rotate 2023-10-31T12:18:17.939 INFO:teuthology.orchestra.run.smithi061.stdout:Rotating logs: 2023-10-31T12:18:17.941 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-31T12:18:17.941 DEBUG:teuthology.orchestra.run.smithi061:> /usr/sbin/getenforce 2023-10-31T12:18:17.965 INFO:teuthology.orchestra.run.smithi061.stdout:Permissive 2023-10-31T12:18:17.965 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi061.front.sepia.ceph.com': 'permissive'} 2023-10-31T12:18:17.966 DEBUG:teuthology.orchestra.run.smithi061:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-31T12:18:18.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-31T12:18:18.033 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-31T12:18:18.034 DEBUG:teuthology.orchestra.run.smithi061:> sudo /usr/sbin/setenforce permissive 2023-10-31T12:18:18.108 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-31T12:18:18.118 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-31T12:18:18.124 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-31T12:18:18.242 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-31T12:18:18.258 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-31T12:18:18.259 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi061.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-31T12:23:53.100 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi061.front.sepia.ceph.com')] 2023-10-31T12:23:53.101 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi061.front.sepia.ceph.com' 2023-10-31T12:23:53.102 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-31T12:23:53.200 DEBUG:teuthology.orchestra.run.smithi061:> true 2023-10-31T12:23:53.269 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi061.front.sepia.ceph.com' 2023-10-31T12:23:53.269 INFO:teuthology.run_tasks:Running task clock... 2023-10-31T12:23:53.286 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-31T12:23:53.287 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-31T12:23:53.287 DEBUG:teuthology.orchestra.run.smithi061:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-31T12:23:53.343 INFO:teuthology.orchestra.run.smithi061.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-31T12:23:53.363 INFO:teuthology.orchestra.run.smithi061.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-31T12:23:53.401 INFO:teuthology.orchestra.run.smithi061.stderr:sudo: ntpd: command not found 2023-10-31T12:23:53.418 INFO:teuthology.orchestra.run.smithi061.stdout:506 Cannot talk to daemon 2023-10-31T12:23:53.439 INFO:teuthology.orchestra.run.smithi061.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-31T12:23:53.460 INFO:teuthology.orchestra.run.smithi061.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-31T12:23:53.508 INFO:teuthology.orchestra.run.smithi061.stderr:bash: line 1: ntpq: command not found 2023-10-31T12:23:53.511 INFO:teuthology.orchestra.run.smithi061.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-31T12:23:53.511 INFO:teuthology.orchestra.run.smithi061.stdout:=============================================================================== 2023-10-31T12:23:53.511 INFO:teuthology.orchestra.run.smithi061.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-31T12:23:53.512 INFO:teuthology.orchestra.run.smithi061.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-31T12:23:53.512 INFO:teuthology.orchestra.run.smithi061.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-31T12:23:53.512 INFO:teuthology.orchestra.run.smithi061.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-31T12:23:53.514 INFO:teuthology.run_tasks:Running task install... 2023-10-31T12:23:53.522 DEBUG:teuthology.task.install:project ceph 2023-10-31T12:23:53.522 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': 'e83bf4821bfb604694b03a67d7390bfb38655506'}} 2023-10-31T12:23:53.523 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': 'e83bf4821bfb604694b03a67d7390bfb38655506'} 2023-10-31T12:23:53.523 INFO:teuthology.task.install:Using flavor: crimson 2023-10-31T12:23:53.530 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-31T12:23:53.530 INFO:teuthology.task.install:extra packages: [] 2023-10-31T12:23:53.531 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e83bf4821bfb604694b03a67d7390bfb38655506', 'tag': None, 'wait_for_package': False} 2023-10-31T12:23:53.531 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=e83bf4821bfb604694b03a67d7390bfb38655506 2023-10-31T12:23:53.701 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-wip-matanb-crimson-do_osd_ops_execute-v3/e83bf4821bfb604694b03a67d7390bfb38655506/centos/9/flavors/crimson/ 2023-10-31T12:23:53.701 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6792.ge83bf482 2023-10-31T12:23:53.831 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-wip-matanb-crimson-do_osd_ops_execute-v3/e83bf4821bfb604694b03a67d7390bfb38655506/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-wip-matanb-crimson-do_osd_ops_execute-v3/e83bf4821bfb604694b03a67d7390bfb38655506/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-wip-matanb-crimson-do_osd_ops_execute-v3/e83bf4821bfb604694b03a67d7390bfb38655506/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-31T12:23:53.832 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:23:53.832 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-31T12:23:53.865 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-31T12:23:53.865 DEBUG:teuthology.orchestra.run.smithi061:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e83bf4821bfb604694b03a67d7390bfb38655506/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-31T12:23:53.937 DEBUG:teuthology.orchestra.run.smithi061:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-31T12:23:54.024 DEBUG:teuthology.orchestra.run.smithi061:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-31T12:23:54.054 INFO:teuthology.orchestra.run.smithi061.stdout:check_obsoletes = 1 2023-10-31T12:23:54.055 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum clean all 2023-10-31T12:23:54.311 INFO:teuthology.orchestra.run.smithi061.stdout:72 files removed 2023-10-31T12:23:54.339 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-radosgw 2023-10-31T12:23:55.037 INFO:teuthology.orchestra.run.smithi061.stdout:ceph packages for x86_64 224 kB/s | 81 kB 00:00 2023-10-31T12:23:55.355 INFO:teuthology.orchestra.run.smithi061.stdout:ceph noarch packages 35 kB/s | 9.4 kB 00:00 2023-10-31T12:23:55.697 INFO:teuthology.orchestra.run.smithi061.stdout:ceph source packages 6.4 kB/s | 1.9 kB 00:00 2023-10-31T12:23:56.099 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS-9 - AppStream 17 MB/s | 6.3 MB 00:00 2023-10-31T12:23:57.467 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2023-10-31T12:23:59.210 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 9 - BaseOS 5.6 MB/s | 7.9 MB 00:01 2023-10-31T12:24:20.540 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 9 - AppStream 905 kB/s | 18 MB 00:20 2023-10-31T12:24:24.674 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 9 - CRB 9.1 MB/s | 5.8 MB 00:00 2023-10-31T12:24:26.333 INFO:teuthology.orchestra.run.smithi061.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2023-10-31T12:24:27.318 INFO:teuthology.orchestra.run.smithi061.stdout:Extra Packages for Enterprise Linux 22 MB/s | 19 MB 00:00 2023-10-31T12:24:31.977 INFO:teuthology.orchestra.run.smithi061.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2023-10-31T12:24:34.463 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:24:34.474 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:24:34.475 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-10-31T12:24:34.475 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:24:34.475 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-10-31T12:24:34.476 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-radosgw x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 8.4 M 2023-10-31T12:24:34.476 INFO:teuthology.orchestra.run.smithi061.stdout:Upgrading: 2023-10-31T12:24:34.476 INFO:teuthology.orchestra.run.smithi061.stdout: librados2 x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 3.1 M 2023-10-31T12:24:34.476 INFO:teuthology.orchestra.run.smithi061.stdout: librbd1 x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 3.0 M 2023-10-31T12:24:34.476 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-10-31T12:24:34.477 INFO:teuthology.orchestra.run.smithi061.stdout: boost-program-options 2023-10-31T12:24:34.477 INFO:teuthology.orchestra.run.smithi061.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-31T12:24:34.477 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-base x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 6.6 M 2023-10-31T12:24:34.477 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-common x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 17 M 2023-10-31T12:24:34.477 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-selinux x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 24 k 2023-10-31T12:24:34.478 INFO:teuthology.orchestra.run.smithi061.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-31T12:24:34.478 INFO:teuthology.orchestra.run.smithi061.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-31T12:24:34.478 INFO:teuthology.orchestra.run.smithi061.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-31T12:24:34.478 INFO:teuthology.orchestra.run.smithi061.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-31T12:24:34.478 INFO:teuthology.orchestra.run.smithi061.stdout: libcephfs2 x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 727 k 2023-10-31T12:24:34.479 INFO:teuthology.orchestra.run.smithi061.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-31T12:24:34.479 INFO:teuthology.orchestra.run.smithi061.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-31T12:24:34.479 INFO:teuthology.orchestra.run.smithi061.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-31T12:24:34.479 INFO:teuthology.orchestra.run.smithi061.stdout: libradosstriper1 x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 469 k 2023-10-31T12:24:34.479 INFO:teuthology.orchestra.run.smithi061.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-31T12:24:34.480 INFO:teuthology.orchestra.run.smithi061.stdout: librgw2 x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 4.5 M 2023-10-31T12:24:34.480 INFO:teuthology.orchestra.run.smithi061.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-31T12:24:34.480 INFO:teuthology.orchestra.run.smithi061.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-31T12:24:34.480 INFO:teuthology.orchestra.run.smithi061.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-31T12:24:34.480 INFO:teuthology.orchestra.run.smithi061.stdout: python3-ceph-argparse 2023-10-31T12:24:34.481 INFO:teuthology.orchestra.run.smithi061.stdout: x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 45 k 2023-10-31T12:24:34.481 INFO:teuthology.orchestra.run.smithi061.stdout: python3-ceph-common x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 127 k 2023-10-31T12:24:34.481 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cephfs x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 162 k 2023-10-31T12:24:34.481 INFO:teuthology.orchestra.run.smithi061.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-31T12:24:34.482 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rados x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 320 k 2023-10-31T12:24:34.482 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rbd x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 298 k 2023-10-31T12:24:34.482 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rgw x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 99 k 2023-10-31T12:24:34.482 INFO:teuthology.orchestra.run.smithi061.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-31T12:24:34.482 INFO:teuthology.orchestra.run.smithi061.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-31T12:24:34.483 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:24:34.483 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-10-31T12:24:34.483 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:24:34.483 INFO:teuthology.orchestra.run.smithi061.stdout:Install 28 Packages 2023-10-31T12:24:34.483 INFO:teuthology.orchestra.run.smithi061.stdout:Upgrade 2 Packages 2023-10-31T12:24:34.484 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:24:34.484 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 54 M 2023-10-31T12:24:34.484 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-10-31T12:24:35.695 INFO:teuthology.orchestra.run.smithi061.stdout:(1/30): ceph-radosgw-18.0.0-6792.ge83bf482.el9. 8.3 MB/s | 8.4 MB 00:01 2023-10-31T12:24:35.761 INFO:teuthology.orchestra.run.smithi061.stdout:(2/30): ceph-selinux-18.0.0-6792.ge83bf482.el9. 363 kB/s | 24 kB 00:00 2023-10-31T12:24:35.870 INFO:teuthology.orchestra.run.smithi061.stdout:(3/30): libcephfs2-18.0.0-6792.ge83bf482.el9.x8 6.6 MB/s | 727 kB 00:00 2023-10-31T12:24:35.945 INFO:teuthology.orchestra.run.smithi061.stdout:(4/30): libradosstriper1-18.0.0-6792.ge83bf482. 6.1 MB/s | 469 kB 00:00 2023-10-31T12:24:36.112 INFO:teuthology.orchestra.run.smithi061.stdout:(5/30): ceph-common-18.0.0-6792.ge83bf482.el9.x 12 MB/s | 17 MB 00:01 2023-10-31T12:24:36.154 INFO:teuthology.orchestra.run.smithi061.stdout:(6/30): python3-ceph-argparse-18.0.0-6792.ge83b 1.1 MB/s | 45 kB 00:00 2023-10-31T12:24:36.246 INFO:teuthology.orchestra.run.smithi061.stdout:(7/30): librgw2-18.0.0-6792.ge83bf482.el9.x86_6 15 MB/s | 4.5 MB 00:00 2023-10-31T12:24:36.272 INFO:teuthology.orchestra.run.smithi061.stdout:(8/30): python3-ceph-common-18.0.0-6792.ge83bf4 1.1 MB/s | 127 kB 00:00 2023-10-31T12:24:36.297 INFO:teuthology.orchestra.run.smithi061.stdout:(9/30): python3-cephfs-18.0.0-6792.ge83bf482.el 3.2 MB/s | 162 kB 00:00 2023-10-31T12:24:36.322 INFO:teuthology.orchestra.run.smithi061.stdout:(10/30): python3-rados-18.0.0-6792.ge83bf482.el 6.3 MB/s | 320 kB 00:00 2023-10-31T12:24:36.347 INFO:teuthology.orchestra.run.smithi061.stdout:(11/30): python3-rbd-18.0.0-6792.ge83bf482.el9. 5.8 MB/s | 298 kB 00:00 2023-10-31T12:24:36.373 INFO:teuthology.orchestra.run.smithi061.stdout:(12/30): python3-rgw-18.0.0-6792.ge83bf482.el9. 1.9 MB/s | 99 kB 00:00 2023-10-31T12:24:36.481 INFO:teuthology.orchestra.run.smithi061.stdout:(13/30): boost-program-options-1.75.0-8.el9.x86 794 kB/s | 106 kB 00:00 2023-10-31T12:24:36.506 INFO:teuthology.orchestra.run.smithi061.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2023-10-31T12:24:36.540 INFO:teuthology.orchestra.run.smithi061.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2023-10-31T12:24:36.565 INFO:teuthology.orchestra.run.smithi061.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2023-10-31T12:24:36.591 INFO:teuthology.orchestra.run.smithi061.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 825 kB/s | 41 kB 00:00 2023-10-31T12:24:36.692 INFO:teuthology.orchestra.run.smithi061.stdout:(18/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 361 kB/s | 45 kB 00:00 2023-10-31T12:24:36.867 INFO:teuthology.orchestra.run.smithi061.stdout:(19/30): ceph-base-18.0.0-6792.ge83bf482.el9.x8 3.0 MB/s | 6.6 MB 00:02 2023-10-31T12:24:37.042 INFO:teuthology.orchestra.run.smithi061.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 881 kB/s | 309 kB 00:00 2023-10-31T12:24:37.217 INFO:teuthology.orchestra.run.smithi061.stdout:(21/30): fmt-8.1.1-5.el9.x86_64.rpm 177 kB/s | 111 kB 00:00 2023-10-31T12:24:37.301 INFO:teuthology.orchestra.run.smithi061.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 582 kB/s | 49 kB 00:00 2023-10-31T12:24:37.326 INFO:teuthology.orchestra.run.smithi061.stdout:(23/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 86 kB/s | 24 kB 00:00 2023-10-31T12:24:37.426 INFO:teuthology.orchestra.run.smithi061.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 539 kB/s | 67 kB 00:00 2023-10-31T12:24:37.560 INFO:teuthology.orchestra.run.smithi061.stdout:(25/30): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2023-10-31T12:24:37.677 INFO:teuthology.orchestra.run.smithi061.stdout:(26/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 2.3 MB/s | 837 kB 00:00 2023-10-31T12:24:37.802 INFO:teuthology.orchestra.run.smithi061.stdout:(27/30): libarrow-9.0.0-9.el9.x86_64.rpm 4.7 MB/s | 4.4 MB 00:00 2023-10-31T12:24:37.835 INFO:teuthology.orchestra.run.smithi061.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 5.7 MB/s | 1.6 MB 00:00 2023-10-31T12:24:38.111 INFO:teuthology.orchestra.run.smithi061.stdout:(29/30): librbd1-18.0.0-6792.ge83bf482.el9.x86_ 9.8 MB/s | 3.0 MB 00:00 2023-10-31T12:24:38.286 INFO:teuthology.orchestra.run.smithi061.stdout:(30/30): librados2-18.0.0-6792.ge83bf482.el9.x8 5.1 MB/s | 3.1 MB 00:00 2023-10-31T12:24:38.289 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:24:38.290 INFO:teuthology.orchestra.run.smithi061.stdout:Total 14 MB/s | 54 MB 00:03 2023-10-31T12:24:38.302 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-10-31T12:24:38.345 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-10-31T12:24:38.345 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-10-31T12:24:38.836 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-10-31T12:24:38.837 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-10-31T12:24:40.408 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-10-31T12:24:40.767 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-10-31T12:24:40.838 INFO:teuthology.orchestra.run.smithi061.stdout: Upgrading : librados2-2:18.0.0-6792.ge83bf482.el9.x86_64 2/32 2023-10-31T12:24:40.930 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librados2-2:18.0.0-6792.ge83bf482.el9.x86_64 2/32 2023-10-31T12:24:41.029 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-rados-2:18.0.0-6792.ge83bf482.el9.x86_64 3/32 2023-10-31T12:24:41.056 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libcephfs2-2:18.0.0-6792.ge83bf482.el9.x86_64 4/32 2023-10-31T12:24:41.119 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: libcephfs2-2:18.0.0-6792.ge83bf482.el9.x86_64 4/32 2023-10-31T12:24:41.160 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/32 2023-10-31T12:24:41.186 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/32 2023-10-31T12:24:41.215 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/32 2023-10-31T12:24:41.268 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-ceph-argparse-2:18.0.0-6792.ge83bf482.el9. 8/32 2023-10-31T12:24:41.336 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-cephfs-2:18.0.0-6792.ge83bf482.el9.x86_64 9/32 2023-10-31T12:24:41.359 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libradosstriper1-2:18.0.0-6792.ge83bf482.el9.x86_6 10/32 2023-10-31T12:24:41.448 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6792.ge83bf482.el9.x86_6 10/32 2023-10-31T12:24:41.484 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/32 2023-10-31T12:24:41.595 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-10-31T12:24:41.626 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-10-31T12:24:41.667 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-10-31T12:24:41.699 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-10-31T12:24:41.726 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-10-31T12:24:41.761 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-10-31T12:24:41.786 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-31T12:24:42.122 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-10-31T12:24:42.146 INFO:teuthology.orchestra.run.smithi061.stdout: Upgrading : librbd1-2:18.0.0-6792.ge83bf482.el9.x86_64 20/32 2023-10-31T12:24:42.237 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librbd1-2:18.0.0-6792.ge83bf482.el9.x86_64 20/32 2023-10-31T12:24:42.268 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-rbd-2:18.0.0-6792.ge83bf482.el9.x86_64 21/32 2023-10-31T12:24:42.305 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-10-31T12:24:42.759 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-10-31T12:24:42.790 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : librgw2-2:18.0.0-6792.ge83bf482.el9.x86_64 24/32 2023-10-31T12:24:42.845 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librgw2-2:18.0.0-6792.ge83bf482.el9.x86_64 24/32 2023-10-31T12:24:42.924 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-rgw-2:18.0.0-6792.ge83bf482.el9.x86_64 25/32 2023-10-31T12:24:42.960 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-ceph-common-2:18.0.0-6792.ge83bf482.el9.x8 26/32 2023-10-31T12:24:45.055 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-common-2:18.0.0-6792.ge83bf482.el9.x86_64 27/32 2023-10-31T12:24:45.090 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-common-2:18.0.0-6792.ge83bf482.el9.x86_64 27/32 2023-10-31T12:24:45.862 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-common-2:18.0.0-6792.ge83bf482.el9.x86_64 27/32 2023-10-31T12:24:45.902 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-base-2:18.0.0-6792.ge83bf482.el9.x86_64 28/32 2023-10-31T12:24:45.981 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-base-2:18.0.0-6792.ge83bf482.el9.x86_64 28/32 2023-10-31T12:24:45.981 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-31T12:24:45.981 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-31T12:24:45.981 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:24:46.013 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-selinux-2:18.0.0-6792.ge83bf482.el9.x86_64 29/32 2023-10-31T12:24:56.021 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6792.ge83bf482.el9.x86_64 29/32 2023-10-31T12:24:56.021 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /sys 2023-10-31T12:24:56.021 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /proc 2023-10-31T12:24:56.021 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /mnt 2023-10-31T12:24:56.022 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /var/tmp 2023-10-31T12:24:56.022 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /home 2023-10-31T12:24:56.022 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /root 2023-10-31T12:24:56.022 INFO:teuthology.orchestra.run.smithi061.stdout:skipping the directory /tmp 2023-10-31T12:24:56.022 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:24:56.904 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-radosgw-2:18.0.0-6792.ge83bf482.el9.x86_64 30/32 2023-10-31T12:24:56.929 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6792.ge83bf482.el9.x86_64 30/32 2023-10-31T12:24:56.930 INFO:teuthology.orchestra.run.smithi061.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-31T12:24:56.930 INFO:teuthology.orchestra.run.smithi061.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-31T12:24:56.930 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-31T12:24:56.930 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-31T12:24:56.931 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:24:56.931 INFO:teuthology.orchestra.run.smithi061.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-31T12:24:57.007 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-10-31T12:24:57.007 INFO:teuthology.orchestra.run.smithi061.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-31T12:24:58.141 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-10-31T12:24:58.142 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-base-2:18.0.0-6792.ge83bf482.el9.x86_64 1/32 2023-10-31T12:24:58.142 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-common-2:18.0.0-6792.ge83bf482.el9.x86_64 2/32 2023-10-31T12:24:58.142 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-radosgw-2:18.0.0-6792.ge83bf482.el9.x86_64 3/32 2023-10-31T12:24:58.142 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-selinux-2:18.0.0-6792.ge83bf482.el9.x86_64 4/32 2023-10-31T12:24:58.143 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libcephfs2-2:18.0.0-6792.ge83bf482.el9.x86_64 5/32 2023-10-31T12:24:58.143 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libradosstriper1-2:18.0.0-6792.ge83bf482.el9.x86_6 6/32 2023-10-31T12:24:58.143 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librgw2-2:18.0.0-6792.ge83bf482.el9.x86_64 7/32 2023-10-31T12:24:58.143 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6792.ge83bf482.el9. 8/32 2023-10-31T12:24:58.143 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-ceph-common-2:18.0.0-6792.ge83bf482.el9.x8 9/32 2023-10-31T12:24:58.144 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-cephfs-2:18.0.0-6792.ge83bf482.el9.x86_64 10/32 2023-10-31T12:24:58.144 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-rados-2:18.0.0-6792.ge83bf482.el9.x86_64 11/32 2023-10-31T12:24:58.144 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-rbd-2:18.0.0-6792.ge83bf482.el9.x86_64 12/32 2023-10-31T12:24:58.144 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-rgw-2:18.0.0-6792.ge83bf482.el9.x86_64 13/32 2023-10-31T12:24:58.144 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-10-31T12:24:58.145 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-10-31T12:24:58.145 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-10-31T12:24:58.145 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-10-31T12:24:58.145 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-10-31T12:24:58.146 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-10-31T12:24:58.146 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-10-31T12:24:58.146 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-10-31T12:24:58.146 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-10-31T12:24:58.146 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-10-31T12:24:58.147 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-10-31T12:24:58.147 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-10-31T12:24:58.147 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-10-31T12:24:58.147 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-10-31T12:24:58.147 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-10-31T12:24:58.148 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librados2-2:18.0.0-6792.ge83bf482.el9.x86_64 29/32 2023-10-31T12:24:58.148 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-10-31T12:24:58.148 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librbd1-2:18.0.0-6792.ge83bf482.el9.x86_64 31/32 2023-10-31T12:24:59.264 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-10-31T12:24:59.264 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:24:59.265 INFO:teuthology.orchestra.run.smithi061.stdout:Upgraded: 2023-10-31T12:24:59.265 INFO:teuthology.orchestra.run.smithi061.stdout: librados2-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:24:59.265 INFO:teuthology.orchestra.run.smithi061.stdout: librbd1-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:24:59.265 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-10-31T12:24:59.265 INFO:teuthology.orchestra.run.smithi061.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-31T12:24:59.266 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-base-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:24:59.266 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-common-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:24:59.266 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-radosgw-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:24:59.266 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-selinux-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:24:59.266 INFO:teuthology.orchestra.run.smithi061.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-31T12:24:59.267 INFO:teuthology.orchestra.run.smithi061.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-31T12:24:59.267 INFO:teuthology.orchestra.run.smithi061.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-31T12:24:59.267 INFO:teuthology.orchestra.run.smithi061.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-31T12:24:59.267 INFO:teuthology.orchestra.run.smithi061.stdout: libcephfs2-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:24:59.267 INFO:teuthology.orchestra.run.smithi061.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-31T12:24:59.268 INFO:teuthology.orchestra.run.smithi061.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-31T12:24:59.268 INFO:teuthology.orchestra.run.smithi061.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-31T12:24:59.268 INFO:teuthology.orchestra.run.smithi061.stdout: libradosstriper1-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:24:59.268 INFO:teuthology.orchestra.run.smithi061.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-31T12:24:59.268 INFO:teuthology.orchestra.run.smithi061.stdout: librgw2-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:24:59.269 INFO:teuthology.orchestra.run.smithi061.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-31T12:24:59.269 INFO:teuthology.orchestra.run.smithi061.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-31T12:24:59.269 INFO:teuthology.orchestra.run.smithi061.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-31T12:24:59.269 INFO:teuthology.orchestra.run.smithi061.stdout: python3-ceph-argparse-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:24:59.269 INFO:teuthology.orchestra.run.smithi061.stdout: python3-ceph-common-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:24:59.270 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cephfs-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:24:59.270 INFO:teuthology.orchestra.run.smithi061.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-31T12:24:59.270 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rados-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:24:59.270 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rbd-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:24:59.270 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rgw-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:24:59.271 INFO:teuthology.orchestra.run.smithi061.stdout: re2-1:20211101-3.el9.x86_64 2023-10-31T12:24:59.271 INFO:teuthology.orchestra.run.smithi061.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-31T12:24:59.271 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:24:59.271 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:24:59.535 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-test 2023-10-31T12:25:00.099 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:00:29 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:25:00.248 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:25:00.248 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:25:00.249 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-10-31T12:25:00.249 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:25:00.249 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-10-31T12:25:00.250 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-test x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 117 M 2023-10-31T12:25:00.250 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-10-31T12:25:00.250 INFO:teuthology.orchestra.run.smithi061.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2023-10-31T12:25:00.250 INFO:teuthology.orchestra.run.smithi061.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-31T12:25:00.251 INFO:teuthology.orchestra.run.smithi061.stdout: libasan x86_64 11.4.1-2.3.el9 CentOS-AppStream 418 k 2023-10-31T12:25:00.251 INFO:teuthology.orchestra.run.smithi061.stdout: libcephsqlite x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 161 k 2023-10-31T12:25:00.251 INFO:teuthology.orchestra.run.smithi061.stdout: libubsan x86_64 11.4.1-2.3.el9 CentOS-AppStream 186 k 2023-10-31T12:25:00.251 INFO:teuthology.orchestra.run.smithi061.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-31T12:25:00.251 INFO:teuthology.orchestra.run.smithi061.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-31T12:25:00.252 INFO:teuthology.orchestra.run.smithi061.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-31T12:25:00.252 INFO:teuthology.orchestra.run.smithi061.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2023-10-31T12:25:00.252 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:25:00.252 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-10-31T12:25:00.253 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:25:00.253 INFO:teuthology.orchestra.run.smithi061.stdout:Install 10 Packages 2023-10-31T12:25:00.253 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:25:00.254 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 120 M 2023-10-31T12:25:00.254 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 1.1 G 2023-10-31T12:25:00.254 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-10-31T12:25:01.061 INFO:teuthology.orchestra.run.smithi061.stdout:(1/10): libasan-11.4.1-2.3.el9.x86_64.rpm 2.4 MB/s | 418 kB 00:00 2023-10-31T12:25:01.094 INFO:teuthology.orchestra.run.smithi061.stdout:(2/10): libubsan-11.4.1-2.3.el9.x86_64.rpm 5.5 MB/s | 186 kB 00:00 2023-10-31T12:25:01.128 INFO:teuthology.orchestra.run.smithi061.stdout:(3/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 6.4 MB/s | 217 kB 00:00 2023-10-31T12:25:01.181 INFO:teuthology.orchestra.run.smithi061.stdout:(4/10): socat-1.7.4.1-5.el9.x86_64.rpm 5.6 MB/s | 300 kB 00:00 2023-10-31T12:25:01.214 INFO:teuthology.orchestra.run.smithi061.stdout:(5/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-10-31T12:25:01.231 INFO:teuthology.orchestra.run.smithi061.stdout:(6/10): libcephsqlite-18.0.0-6792.ge83bf482.el9 470 kB/s | 161 kB 00:00 2023-10-31T12:25:01.365 INFO:teuthology.orchestra.run.smithi061.stdout:(7/10): jq-1.6-15.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2023-10-31T12:25:01.798 INFO:teuthology.orchestra.run.smithi061.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 281 kB/s | 122 kB 00:00 2023-10-31T12:25:01.957 INFO:teuthology.orchestra.run.smithi061.stdout:(9/10): cryptopp-8.6.0-1.el9.x86_64.rpm 1.7 MB/s | 1.3 MB 00:00 2023-10-31T12:25:06.498 INFO:teuthology.orchestra.run.smithi061.stdout:(10/10): ceph-test-18.0.0-6792.ge83bf482.el9.x8 21 MB/s | 117 MB 00:05 2023-10-31T12:25:06.501 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:25:06.501 INFO:teuthology.orchestra.run.smithi061.stdout:Total 19 MB/s | 120 MB 00:06 2023-10-31T12:25:06.516 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-10-31T12:25:06.540 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-10-31T12:25:06.541 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-10-31T12:25:07.407 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-10-31T12:25:07.408 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-10-31T12:25:08.584 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-10-31T12:25:08.669 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2023-10-31T12:25:08.698 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2023-10-31T12:25:08.743 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2023-10-31T12:25:08.782 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2023-10-31T12:25:08.816 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-10-31T12:25:08.847 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : jq-1.6-15.el9.x86_64 6/10 2023-10-31T12:25:08.876 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libubsan-11.4.1-2.3.el9.x86_64 7/10 2023-10-31T12:25:08.918 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libasan-11.4.1-2.3.el9.x86_64 8/10 2023-10-31T12:25:08.944 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libcephsqlite-2:18.0.0-6792.ge83bf482.el9.x86_64 9/10 2023-10-31T12:25:26.155 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6792.ge83bf482.el9.x86_64 9/10 2023-10-31T12:25:26.236 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-test-2:18.0.0-6792.ge83bf482.el9.x86_64 10/10 2023-10-31T12:25:34.173 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-test-2:18.0.0-6792.ge83bf482.el9.x86_64 10/10 2023-10-31T12:25:34.174 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-test-2:18.0.0-6792.ge83bf482.el9.x86_64 1/10 2023-10-31T12:25:34.174 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libcephsqlite-2:18.0.0-6792.ge83bf482.el9.x86_64 2/10 2023-10-31T12:25:34.174 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libasan-11.4.1-2.3.el9.x86_64 3/10 2023-10-31T12:25:34.175 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libubsan-11.4.1-2.3.el9.x86_64 4/10 2023-10-31T12:25:34.175 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-10-31T12:25:34.175 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 6/10 2023-10-31T12:25:34.175 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 7/10 2023-10-31T12:25:34.175 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : jq-1.6-15.el9.x86_64 8/10 2023-10-31T12:25:34.176 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2023-10-31T12:25:34.671 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2023-10-31T12:25:34.671 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:25:34.671 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-10-31T12:25:34.672 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-test-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:25:34.672 INFO:teuthology.orchestra.run.smithi061.stdout: cryptopp-8.6.0-1.el9.x86_64 2023-10-31T12:25:34.672 INFO:teuthology.orchestra.run.smithi061.stdout: jq-1.6-15.el9.x86_64 2023-10-31T12:25:34.672 INFO:teuthology.orchestra.run.smithi061.stdout: libasan-11.4.1-2.3.el9.x86_64 2023-10-31T12:25:34.673 INFO:teuthology.orchestra.run.smithi061.stdout: libcephsqlite-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:25:34.673 INFO:teuthology.orchestra.run.smithi061.stdout: libubsan-11.4.1-2.3.el9.x86_64 2023-10-31T12:25:34.673 INFO:teuthology.orchestra.run.smithi061.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-31T12:25:34.673 INFO:teuthology.orchestra.run.smithi061.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-31T12:25:34.673 INFO:teuthology.orchestra.run.smithi061.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-31T12:25:34.674 INFO:teuthology.orchestra.run.smithi061.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2023-10-31T12:25:34.674 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:25:34.674 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:25:34.986 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph 2023-10-31T12:25:35.562 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:04 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:25:35.710 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:25:35.713 INFO:teuthology.orchestra.run.smithi061.stdout:========================================================================================= 2023-10-31T12:25:35.713 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-10-31T12:25:35.713 INFO:teuthology.orchestra.run.smithi061.stdout:========================================================================================= 2023-10-31T12:25:35.713 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-10-31T12:25:35.714 INFO:teuthology.orchestra.run.smithi061.stdout: ceph x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 6.5 k 2023-10-31T12:25:35.714 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-10-31T12:25:35.714 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mds x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 2.0 M 2023-10-31T12:25:35.715 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 1.5 M 2023-10-31T12:25:35.715 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6792.ge83bf482.el9 ceph-noarch 242 k 2023-10-31T12:25:35.715 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mon x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 8.8 M 2023-10-31T12:25:35.715 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-osd x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 116 M 2023-10-31T12:25:35.716 INFO:teuthology.orchestra.run.smithi061.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-31T12:25:35.716 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-31T12:25:35.716 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-31T12:25:35.716 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-31T12:25:35.717 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-31T12:25:35.717 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-31T12:25:35.717 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-31T12:25:35.717 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-31T12:25:35.718 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-31T12:25:35.718 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-31T12:25:35.718 INFO:teuthology.orchestra.run.smithi061.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-31T12:25:35.718 INFO:teuthology.orchestra.run.smithi061.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-31T12:25:35.719 INFO:teuthology.orchestra.run.smithi061.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-31T12:25:35.719 INFO:teuthology.orchestra.run.smithi061.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-31T12:25:35.719 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-31T12:25:35.719 INFO:teuthology.orchestra.run.smithi061.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-31T12:25:35.719 INFO:teuthology.orchestra.run.smithi061.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-31T12:25:35.720 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-31T12:25:35.720 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-31T12:25:35.720 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-31T12:25:35.720 INFO:teuthology.orchestra.run.smithi061.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-31T12:25:35.721 INFO:teuthology.orchestra.run.smithi061.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-31T12:25:35.721 INFO:teuthology.orchestra.run.smithi061.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-31T12:25:35.721 INFO:teuthology.orchestra.run.smithi061.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-10-31T12:25:35.721 INFO:teuthology.orchestra.run.smithi061.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-31T12:25:35.722 INFO:teuthology.orchestra.run.smithi061.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-31T12:25:35.722 INFO:teuthology.orchestra.run.smithi061.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-31T12:25:35.722 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:25:35.722 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-10-31T12:25:35.723 INFO:teuthology.orchestra.run.smithi061.stdout:========================================================================================= 2023-10-31T12:25:35.723 INFO:teuthology.orchestra.run.smithi061.stdout:Install 33 Packages 2023-10-31T12:25:35.723 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:25:35.724 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 133 M 2023-10-31T12:25:35.724 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 1.4 G 2023-10-31T12:25:35.725 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-10-31T12:25:36.125 INFO:teuthology.orchestra.run.smithi061.stdout:(1/33): ceph-18.0.0-6792.ge83bf482.el9.x86_64.r 38 kB/s | 6.5 kB 00:00 2023-10-31T12:25:36.375 INFO:teuthology.orchestra.run.smithi061.stdout:(2/33): ceph-mds-18.0.0-6792.ge83bf482.el9.x86_ 4.7 MB/s | 2.0 MB 00:00 2023-10-31T12:25:36.409 INFO:teuthology.orchestra.run.smithi061.stdout:(3/33): ceph-mgr-18.0.0-6792.ge83bf482.el9.x86_ 3.2 MB/s | 1.5 MB 00:00 2023-10-31T12:25:36.501 INFO:teuthology.orchestra.run.smithi061.stdout:(4/33): ceph-mgr-modules-core-18.0.0-6792.ge83b 2.6 MB/s | 242 kB 00:00 2023-10-31T12:25:36.593 INFO:teuthology.orchestra.run.smithi061.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.5 MB/s | 144 kB 00:00 2023-10-31T12:25:36.618 INFO:teuthology.orchestra.run.smithi061.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2023-10-31T12:25:36.643 INFO:teuthology.orchestra.run.smithi061.stdout:(7/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2023-10-31T12:25:36.685 INFO:teuthology.orchestra.run.smithi061.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2023-10-31T12:25:36.719 INFO:teuthology.orchestra.run.smithi061.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-10-31T12:25:36.744 INFO:teuthology.orchestra.run.smithi061.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 4.9 MB/s | 124 kB 00:00 2023-10-31T12:25:36.778 INFO:teuthology.orchestra.run.smithi061.stdout:(11/33): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-10-31T12:25:36.820 INFO:teuthology.orchestra.run.smithi061.stdout:(12/33): python3-urllib3-1.26.5-4.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2023-10-31T12:25:37.003 INFO:teuthology.orchestra.run.smithi061.stdout:(13/33): python3-cryptography-36.0.1-4.el9.x86_ 6.7 MB/s | 1.2 MB 00:00 2023-10-31T12:25:37.178 INFO:teuthology.orchestra.run.smithi061.stdout:(14/33): ceph-mon-18.0.0-6792.ge83bf482.el9.x86 8.4 MB/s | 8.8 MB 00:01 2023-10-31T12:25:37.195 INFO:teuthology.orchestra.run.smithi061.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 266 kB/s | 51 kB 00:00 2023-10-31T12:25:37.671 INFO:teuthology.orchestra.run.smithi061.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 88 kB/s | 43 kB 00:00 2023-10-31T12:25:37.896 INFO:teuthology.orchestra.run.smithi061.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 245 kB/s | 172 kB 00:00 2023-10-31T12:25:38.038 INFO:teuthology.orchestra.run.smithi061.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 76 kB/s | 11 kB 00:00 2023-10-31T12:25:38.088 INFO:teuthology.orchestra.run.smithi061.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 859 kB/s | 358 kB 00:00 2023-10-31T12:25:38.155 INFO:teuthology.orchestra.run.smithi061.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 151 kB/s | 18 kB 00:00 2023-10-31T12:25:38.222 INFO:teuthology.orchestra.run.smithi061.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 173 kB/s | 23 kB 00:00 2023-10-31T12:25:38.272 INFO:teuthology.orchestra.run.smithi061.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 166 kB/s | 19 kB 00:00 2023-10-31T12:25:38.331 INFO:teuthology.orchestra.run.smithi061.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2023-10-31T12:25:38.406 INFO:teuthology.orchestra.run.smithi061.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 346 kB/s | 46 kB 00:00 2023-10-31T12:25:38.465 INFO:teuthology.orchestra.run.smithi061.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 591 kB/s | 79 kB 00:00 2023-10-31T12:25:38.590 INFO:teuthology.orchestra.run.smithi061.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 131 kB/s | 16 kB 00:00 2023-10-31T12:25:38.648 INFO:teuthology.orchestra.run.smithi061.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.1 MB/s | 272 kB 00:00 2023-10-31T12:25:38.724 INFO:teuthology.orchestra.run.smithi061.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 674 kB/s | 90 kB 00:00 2023-10-31T12:25:38.774 INFO:teuthology.orchestra.run.smithi061.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 285 kB/s | 36 kB 00:00 2023-10-31T12:25:38.908 INFO:teuthology.orchestra.run.smithi061.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2023-10-31T12:25:38.949 INFO:teuthology.orchestra.run.smithi061.stdout:(31/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.4 MB/s | 427 kB 00:00 2023-10-31T12:25:39.025 INFO:teuthology.orchestra.run.smithi061.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 171 kB/s | 20 kB 00:00 2023-10-31T12:25:41.641 INFO:teuthology.orchestra.run.smithi061.stdout:(33/33): ceph-osd-18.0.0-6792.ge83bf482.el9.x86 22 MB/s | 116 MB 00:05 2023-10-31T12:25:41.647 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:25:41.648 INFO:teuthology.orchestra.run.smithi061.stdout:Total 22 MB/s | 133 MB 00:05 2023-10-31T12:25:41.682 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-10-31T12:25:41.725 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-10-31T12:25:41.725 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-10-31T12:25:42.748 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-10-31T12:25:42.748 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-10-31T12:25:44.623 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-10-31T12:25:44.677 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-10-31T12:25:44.710 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-10-31T12:25:44.743 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-10-31T12:25:44.838 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-10-31T12:25:44.915 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-10-31T12:25:44.963 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-10-31T12:25:45.000 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-10-31T12:25:45.062 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-10-31T12:25:45.122 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 9/33 2023-10-31T12:25:45.160 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-10-31T12:25:45.211 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-10-31T12:25:45.274 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-31T12:25:45.480 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-10-31T12:25:45.529 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-10-31T12:25:45.577 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-10-31T12:25:45.618 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-10-31T12:25:45.648 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-10-31T12:25:45.727 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-10-31T12:25:45.758 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-10-31T12:25:45.791 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-10-31T12:25:45.833 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-10-31T12:25:45.868 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-10-31T12:25:45.959 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-10-31T12:25:46.008 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-10-31T12:25:46.089 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-10-31T12:25:46.229 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-10-31T12:25:46.365 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-31T12:25:46.530 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6792.ge83bf482.el9. 28/33 2023-10-31T12:25:46.561 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mgr-2:18.0.0-6792.ge83bf482.el9.x86_64 29/33 2023-10-31T12:25:46.586 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6792.ge83bf482.el9.x86_64 29/33 2023-10-31T12:25:46.587 INFO:teuthology.orchestra.run.smithi061.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-31T12:25:46.587 INFO:teuthology.orchestra.run.smithi061.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-31T12:25:46.587 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-31T12:25:46.587 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-31T12:25:46.587 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:05.852 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-osd-2:18.0.0-6792.ge83bf482.el9.x86_64 30/33 2023-10-31T12:26:05.878 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-osd-2:18.0.0-6792.ge83bf482.el9.x86_64 30/33 2023-10-31T12:26:05.879 INFO:teuthology.orchestra.run.smithi061.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-31T12:26:05.879 INFO:teuthology.orchestra.run.smithi061.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-31T12:26:05.879 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-31T12:26:05.879 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-31T12:26:05.880 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:06.802 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mon-2:18.0.0-6792.ge83bf482.el9.x86_64 31/33 2023-10-31T12:26:06.826 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mon-2:18.0.0-6792.ge83bf482.el9.x86_64 31/33 2023-10-31T12:26:06.826 INFO:teuthology.orchestra.run.smithi061.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-31T12:26:06.826 INFO:teuthology.orchestra.run.smithi061.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-31T12:26:06.827 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-31T12:26:06.827 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-31T12:26:06.827 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:07.036 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mds-2:18.0.0-6792.ge83bf482.el9.x86_64 32/33 2023-10-31T12:26:07.060 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mds-2:18.0.0-6792.ge83bf482.el9.x86_64 32/33 2023-10-31T12:26:07.060 INFO:teuthology.orchestra.run.smithi061.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-31T12:26:07.060 INFO:teuthology.orchestra.run.smithi061.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-31T12:26:07.061 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-31T12:26:07.061 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-31T12:26:07.061 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:07.127 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-2:18.0.0-6792.ge83bf482.el9.x86_64 33/33 2023-10-31T12:26:16.852 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-2:18.0.0-6792.ge83bf482.el9.x86_64 33/33 2023-10-31T12:26:16.852 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-2:18.0.0-6792.ge83bf482.el9.x86_64 1/33 2023-10-31T12:26:16.853 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mds-2:18.0.0-6792.ge83bf482.el9.x86_64 2/33 2023-10-31T12:26:16.853 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mgr-2:18.0.0-6792.ge83bf482.el9.x86_64 3/33 2023-10-31T12:26:16.853 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mon-2:18.0.0-6792.ge83bf482.el9.x86_64 4/33 2023-10-31T12:26:16.853 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-osd-2:18.0.0-6792.ge83bf482.el9.x86_64 5/33 2023-10-31T12:26:16.853 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6792.ge83bf482.el9. 6/33 2023-10-31T12:26:16.854 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-10-31T12:26:16.854 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-10-31T12:26:16.854 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-10-31T12:26:16.854 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-10-31T12:26:16.854 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-10-31T12:26:16.855 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-10-31T12:26:16.855 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-10-31T12:26:16.855 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 14/33 2023-10-31T12:26:16.855 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-10-31T12:26:16.855 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-10-31T12:26:16.856 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-10-31T12:26:16.856 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-10-31T12:26:16.856 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-10-31T12:26:16.856 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-10-31T12:26:16.856 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-10-31T12:26:16.857 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-10-31T12:26:16.857 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-10-31T12:26:16.857 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-10-31T12:26:16.857 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-10-31T12:26:16.857 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-10-31T12:26:16.857 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-10-31T12:26:16.858 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-10-31T12:26:16.858 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-10-31T12:26:16.858 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-10-31T12:26:16.858 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-10-31T12:26:16.858 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-10-31T12:26:17.711 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-10-31T12:26:17.711 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:17.711 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-10-31T12:26:17.711 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:26:17.712 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mds-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:26:17.712 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:26:17.712 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-modules-core-2:18.0.0-6792.ge83bf482.el9.noarch 2023-10-31T12:26:17.712 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mon-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:26:17.712 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-osd-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:26:17.713 INFO:teuthology.orchestra.run.smithi061.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-31T12:26:17.713 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-31T12:26:17.713 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-31T12:26:17.713 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-31T12:26:17.713 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-31T12:26:17.713 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-31T12:26:17.714 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-31T12:26:17.714 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-31T12:26:17.714 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-31T12:26:17.714 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-31T12:26:17.714 INFO:teuthology.orchestra.run.smithi061.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-31T12:26:17.715 INFO:teuthology.orchestra.run.smithi061.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-31T12:26:17.715 INFO:teuthology.orchestra.run.smithi061.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-31T12:26:17.715 INFO:teuthology.orchestra.run.smithi061.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-31T12:26:17.715 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-31T12:26:17.715 INFO:teuthology.orchestra.run.smithi061.stdout: python3-ply-3.11-14.el9.noarch 2023-10-31T12:26:17.716 INFO:teuthology.orchestra.run.smithi061.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-31T12:26:17.716 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-31T12:26:17.716 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-31T12:26:17.716 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-31T12:26:17.716 INFO:teuthology.orchestra.run.smithi061.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-31T12:26:17.717 INFO:teuthology.orchestra.run.smithi061.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-31T12:26:17.717 INFO:teuthology.orchestra.run.smithi061.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-31T12:26:17.717 INFO:teuthology.orchestra.run.smithi061.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-10-31T12:26:17.717 INFO:teuthology.orchestra.run.smithi061.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-31T12:26:17.717 INFO:teuthology.orchestra.run.smithi061.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-31T12:26:17.718 INFO:teuthology.orchestra.run.smithi061.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-31T12:26:17.718 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:17.718 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:26:17.933 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-base 2023-10-31T12:26:18.502 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:47 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:26:18.598 INFO:teuthology.orchestra.run.smithi061.stdout:Package ceph-base-2:18.0.0-6792.ge83bf482.el9.x86_64 is already installed. 2023-10-31T12:26:18.650 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:26:18.652 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-10-31T12:26:18.653 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:26:18.710 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install cephadm 2023-10-31T12:26:19.278 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:48 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:26:19.425 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:26:19.427 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:26:19.427 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-10-31T12:26:19.428 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:26:19.428 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-10-31T12:26:19.428 INFO:teuthology.orchestra.run.smithi061.stdout: cephadm noarch 2:18.0.0-6792.ge83bf482.el9 ceph-noarch 270 k 2023-10-31T12:26:19.429 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:19.429 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-10-31T12:26:19.429 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:26:19.430 INFO:teuthology.orchestra.run.smithi061.stdout:Install 1 Package 2023-10-31T12:26:19.430 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:19.430 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 270 k 2023-10-31T12:26:19.430 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 265 k 2023-10-31T12:26:19.431 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-10-31T12:26:20.033 INFO:teuthology.orchestra.run.smithi061.stdout:cephadm-18.0.0-6792.ge83bf482.el9.noarch.rpm 445 kB/s | 270 kB 00:00 2023-10-31T12:26:20.034 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:26:20.035 INFO:teuthology.orchestra.run.smithi061.stdout:Total 444 kB/s | 270 kB 00:00 2023-10-31T12:26:20.035 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-10-31T12:26:20.040 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-10-31T12:26:20.040 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-10-31T12:26:20.043 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-10-31T12:26:20.044 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-10-31T12:26:20.116 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-10-31T12:26:20.545 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: cephadm-2:18.0.0-6792.ge83bf482.el9.noarch 1/1 2023-10-31T12:26:20.633 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : cephadm-2:18.0.0-6792.ge83bf482.el9.noarch 1/1 2023-10-31T12:26:21.001 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: cephadm-2:18.0.0-6792.ge83bf482.el9.noarch 1/1 2023-10-31T12:26:21.300 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : cephadm-2:18.0.0-6792.ge83bf482.el9.noarch 1/1 2023-10-31T12:26:21.300 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:21.300 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-10-31T12:26:21.300 INFO:teuthology.orchestra.run.smithi061.stdout: cephadm-2:18.0.0-6792.ge83bf482.el9.noarch 2023-10-31T12:26:21.301 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:21.301 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:26:21.490 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-immutable-object-cache 2023-10-31T12:26:22.066 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:51 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:26:22.215 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:26:22.216 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:26:22.216 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repo Size 2023-10-31T12:26:22.217 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:26:22.217 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-10-31T12:26:22.217 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 144 k 2023-10-31T12:26:22.217 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:22.217 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-10-31T12:26:22.217 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:26:22.218 INFO:teuthology.orchestra.run.smithi061.stdout:Install 1 Package 2023-10-31T12:26:22.218 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:22.218 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 144 k 2023-10-31T12:26:22.218 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 437 k 2023-10-31T12:26:22.218 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-10-31T12:26:22.788 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-immutable-object-cache-18.0.0-6792.ge83bf4 253 kB/s | 144 kB 00:00 2023-10-31T12:26:22.789 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:26:22.789 INFO:teuthology.orchestra.run.smithi061.stdout:Total 252 kB/s | 144 kB 00:00 2023-10-31T12:26:22.789 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-10-31T12:26:22.797 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-10-31T12:26:22.797 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-10-31T12:26:22.852 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-10-31T12:26:22.853 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-10-31T12:26:23.007 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-10-31T12:26:23.055 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6792.ge83bf482. 1/1 2023-10-31T12:26:23.081 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6792.ge83bf482. 1/1 2023-10-31T12:26:23.081 INFO:teuthology.orchestra.run.smithi061.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-31T12:26:23.081 INFO:teuthology.orchestra.run.smithi061.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-31T12:26:23.082 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:24.048 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6792.ge83bf482. 1/1 2023-10-31T12:26:24.049 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:24.049 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-10-31T12:26:24.049 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-immutable-object-cache-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:26:24.049 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:24.050 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:26:24.283 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-mgr 2023-10-31T12:26:24.853 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:53 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:26:24.949 INFO:teuthology.orchestra.run.smithi061.stdout:Package ceph-mgr-2:18.0.0-6792.ge83bf482.el9.x86_64 is already installed. 2023-10-31T12:26:25.001 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:26:25.002 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-10-31T12:26:25.002 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:26:25.057 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-mgr-dashboard 2023-10-31T12:26:25.626 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:54 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:26:25.772 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:26:25.773 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:26:25.773 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-10-31T12:26:25.773 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:26:25.773 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-10-31T12:26:25.774 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6792.ge83bf482.el9 ceph-noarch 1.8 M 2023-10-31T12:26:25.774 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-10-31T12:26:25.774 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6792.ge83bf482.el9 ceph-noarch 23 k 2023-10-31T12:26:25.774 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6792.ge83bf482.el9 ceph-noarch 15 k 2023-10-31T12:26:25.774 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-31T12:26:25.775 INFO:teuthology.orchestra.run.smithi061.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-31T12:26:25.775 INFO:teuthology.orchestra.run.smithi061.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-31T12:26:25.775 INFO:teuthology.orchestra.run.smithi061.stdout:Installing weak dependencies: 2023-10-31T12:26:25.775 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-31T12:26:25.775 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:25.776 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-10-31T12:26:25.776 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:26:25.776 INFO:teuthology.orchestra.run.smithi061.stdout:Install 7 Packages 2023-10-31T12:26:25.776 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:25.777 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 2.1 M 2023-10-31T12:26:25.777 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 8.6 M 2023-10-31T12:26:25.777 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-10-31T12:26:26.150 INFO:teuthology.orchestra.run.smithi061.stdout:(1/7): ceph-grafana-dashboards-18.0.0-6792.ge83 135 kB/s | 23 kB 00:00 2023-10-31T12:26:26.192 INFO:teuthology.orchestra.run.smithi061.stdout:(2/7): ceph-prometheus-alerts-18.0.0-6792.ge83b 71 kB/s | 15 kB 00:00 2023-10-31T12:26:26.492 INFO:teuthology.orchestra.run.smithi061.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 26 kB/s | 9.0 kB 00:00 2023-10-31T12:26:26.684 INFO:teuthology.orchestra.run.smithi061.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 160 kB/s | 31 kB 00:00 2023-10-31T12:26:26.726 INFO:teuthology.orchestra.run.smithi061.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 77 kB/s | 41 kB 00:00 2023-10-31T12:26:26.801 INFO:teuthology.orchestra.run.smithi061.stdout:(6/7): ceph-mgr-dashboard-18.0.0-6792.ge83bf482 2.1 MB/s | 1.8 MB 00:00 2023-10-31T12:26:27.059 INFO:teuthology.orchestra.run.smithi061.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 502 kB/s | 188 kB 00:00 2023-10-31T12:26:27.062 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:26:27.063 INFO:teuthology.orchestra.run.smithi061.stdout:Total 1.6 MB/s | 2.1 MB 00:01 2023-10-31T12:26:27.063 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-10-31T12:26:27.072 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-10-31T12:26:27.072 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-10-31T12:26:27.138 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-10-31T12:26:27.138 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-10-31T12:26:27.557 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-10-31T12:26:27.619 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-10-31T12:26:27.653 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-10-31T12:26:27.701 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-10-31T12:26:27.740 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-10-31T12:26:27.769 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6792.ge83bf482.el9.n 5/7 2023-10-31T12:26:28.020 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6792.ge83bf482.el9. 6/7 2023-10-31T12:26:28.046 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6792.ge83bf482.el9.noarc 7/7 2023-10-31T12:26:28.657 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6792.ge83bf482.el9.noarc 7/7 2023-10-31T12:26:28.658 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6792.ge83bf482.el9. 1/7 2023-10-31T12:26:28.658 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6792.ge83bf482.el9.noarc 2/7 2023-10-31T12:26:28.658 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6792.ge83bf482.el9.n 3/7 2023-10-31T12:26:28.658 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-10-31T12:26:28.659 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-10-31T12:26:28.659 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-10-31T12:26:28.990 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-10-31T12:26:28.990 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:28.991 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-10-31T12:26:28.991 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-grafana-dashboards-2:18.0.0-6792.ge83bf482.el9.noarch 2023-10-31T12:26:28.991 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-dashboard-2:18.0.0-6792.ge83bf482.el9.noarch 2023-10-31T12:26:28.991 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-prometheus-alerts-2:18.0.0-6792.ge83bf482.el9.noarch 2023-10-31T12:26:28.992 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-31T12:26:28.992 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-31T12:26:28.992 INFO:teuthology.orchestra.run.smithi061.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-31T12:26:28.992 INFO:teuthology.orchestra.run.smithi061.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-31T12:26:28.992 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:28.993 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:26:29.192 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-31T12:26:29.757 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:01:58 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:26:29.904 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:26:29.905 INFO:teuthology.orchestra.run.smithi061.stdout:============================================================================================ 2023-10-31T12:26:29.905 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-10-31T12:26:29.906 INFO:teuthology.orchestra.run.smithi061.stdout:============================================================================================ 2023-10-31T12:26:29.906 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-10-31T12:26:29.906 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6792.ge83bf482.el9 ceph-noarch 7.4 M 2023-10-31T12:26:29.907 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-10-31T12:26:29.907 INFO:teuthology.orchestra.run.smithi061.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-31T12:26:29.907 INFO:teuthology.orchestra.run.smithi061.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-31T12:26:29.907 INFO:teuthology.orchestra.run.smithi061.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-31T12:26:29.908 INFO:teuthology.orchestra.run.smithi061.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-31T12:26:29.908 INFO:teuthology.orchestra.run.smithi061.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-31T12:26:29.908 INFO:teuthology.orchestra.run.smithi061.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-31T12:26:29.908 INFO:teuthology.orchestra.run.smithi061.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-31T12:26:29.908 INFO:teuthology.orchestra.run.smithi061.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-31T12:26:29.909 INFO:teuthology.orchestra.run.smithi061.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-31T12:26:29.909 INFO:teuthology.orchestra.run.smithi061.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-31T12:26:29.909 INFO:teuthology.orchestra.run.smithi061.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-31T12:26:29.909 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:29.909 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-10-31T12:26:29.910 INFO:teuthology.orchestra.run.smithi061.stdout:============================================================================================ 2023-10-31T12:26:29.910 INFO:teuthology.orchestra.run.smithi061.stdout:Install 12 Packages 2023-10-31T12:26:29.910 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:29.911 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 37 M 2023-10-31T12:26:29.911 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 209 M 2023-10-31T12:26:29.911 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-10-31T12:26:30.032 INFO:teuthology.orchestra.run.smithi061.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 254 kB/s | 32 kB 00:00 2023-10-31T12:26:30.066 INFO:teuthology.orchestra.run.smithi061.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 515 kB/s | 17 kB 00:00 2023-10-31T12:26:30.100 INFO:teuthology.orchestra.run.smithi061.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.1 MB/s | 34 kB 00:00 2023-10-31T12:26:30.158 INFO:teuthology.orchestra.run.smithi061.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2023-10-31T12:26:30.258 INFO:teuthology.orchestra.run.smithi061.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2023-10-31T12:26:30.284 INFO:teuthology.orchestra.run.smithi061.stdout:(6/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 205 kB 00:00 2023-10-31T12:26:30.359 INFO:teuthology.orchestra.run.smithi061.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 48 MB/s | 4.8 MB 00:00 2023-10-31T12:26:30.384 INFO:teuthology.orchestra.run.smithi061.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2023-10-31T12:26:30.464 INFO:teuthology.orchestra.run.smithi061.stdout:(9/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 10 MB/s | 803 kB 00:00 2023-10-31T12:26:30.635 INFO:teuthology.orchestra.run.smithi061.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 57 MB/s | 16 MB 00:00 2023-10-31T12:26:30.660 INFO:teuthology.orchestra.run.smithi061.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2023-10-31T12:26:31.768 INFO:teuthology.orchestra.run.smithi061.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-6 4.0 MB/s | 7.4 MB 00:01 2023-10-31T12:26:31.771 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:26:31.772 INFO:teuthology.orchestra.run.smithi061.stdout:Total 20 MB/s | 37 MB 00:01 2023-10-31T12:26:31.773 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-10-31T12:26:31.799 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-10-31T12:26:31.799 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-10-31T12:26:32.256 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-10-31T12:26:32.256 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-10-31T12:26:33.113 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-10-31T12:26:33.175 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-10-31T12:26:33.206 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-10-31T12:26:33.306 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-10-31T12:26:33.351 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-10-31T12:26:33.537 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-31T12:26:33.566 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-31T12:26:33.660 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-10-31T12:26:34.347 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-10-31T12:26:34.467 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-10-31T12:26:35.785 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-10-31T12:26:36.870 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-10-31T12:26:36.898 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6792.ge83bf 12/12 2023-10-31T12:26:39.393 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6792.ge83bf 12/12 2023-10-31T12:26:39.393 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6792.ge83bf 1/12 2023-10-31T12:26:39.393 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-10-31T12:26:39.394 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-10-31T12:26:39.394 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-10-31T12:26:39.394 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-10-31T12:26:39.394 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-10-31T12:26:39.394 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-10-31T12:26:39.395 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-10-31T12:26:39.395 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-10-31T12:26:39.395 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-10-31T12:26:39.395 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-10-31T12:26:39.934 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-10-31T12:26:39.935 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:39.935 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-10-31T12:26:39.935 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6792.ge83bf482.el9.noarch 2023-10-31T12:26:39.935 INFO:teuthology.orchestra.run.smithi061.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-31T12:26:39.936 INFO:teuthology.orchestra.run.smithi061.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-31T12:26:39.936 INFO:teuthology.orchestra.run.smithi061.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-31T12:26:39.936 INFO:teuthology.orchestra.run.smithi061.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-31T12:26:39.936 INFO:teuthology.orchestra.run.smithi061.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-31T12:26:39.936 INFO:teuthology.orchestra.run.smithi061.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-31T12:26:39.937 INFO:teuthology.orchestra.run.smithi061.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-31T12:26:39.937 INFO:teuthology.orchestra.run.smithi061.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-31T12:26:39.937 INFO:teuthology.orchestra.run.smithi061.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-31T12:26:39.937 INFO:teuthology.orchestra.run.smithi061.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-31T12:26:39.937 INFO:teuthology.orchestra.run.smithi061.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-31T12:26:39.938 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:39.938 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:26:40.271 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-mgr-rook 2023-10-31T12:26:40.845 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:09 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:26:40.994 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:26:40.995 INFO:teuthology.orchestra.run.smithi061.stdout:======================================================================================== 2023-10-31T12:26:40.996 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-10-31T12:26:40.996 INFO:teuthology.orchestra.run.smithi061.stdout:======================================================================================== 2023-10-31T12:26:40.996 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-10-31T12:26:40.996 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-rook noarch 2:18.0.0-6792.ge83bf482.el9 ceph-noarch 48 k 2023-10-31T12:26:40.997 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-10-31T12:26:40.997 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-31T12:26:40.997 INFO:teuthology.orchestra.run.smithi061.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-31T12:26:40.997 INFO:teuthology.orchestra.run.smithi061.stdout: python3-google-auth noarch 1:2.23.3-1.el9 epel 197 k 2023-10-31T12:26:40.998 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-31T12:26:40.998 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-31T12:26:40.998 INFO:teuthology.orchestra.run.smithi061.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-31T12:26:40.998 INFO:teuthology.orchestra.run.smithi061.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-31T12:26:40.998 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-31T12:26:40.999 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-31T12:26:40.999 INFO:teuthology.orchestra.run.smithi061.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-31T12:26:40.999 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-31T12:26:40.999 INFO:teuthology.orchestra.run.smithi061.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-31T12:26:40.999 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:41.000 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-10-31T12:26:41.000 INFO:teuthology.orchestra.run.smithi061.stdout:======================================================================================== 2023-10-31T12:26:41.000 INFO:teuthology.orchestra.run.smithi061.stdout:Install 13 Packages 2023-10-31T12:26:41.000 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:41.001 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 2.0 M 2023-10-31T12:26:41.001 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 26 M 2023-10-31T12:26:41.001 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-10-31T12:26:41.173 INFO:teuthology.orchestra.run.smithi061.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 199 kB/s | 24 kB 00:00 2023-10-31T12:26:41.199 INFO:teuthology.orchestra.run.smithi061.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 122 kB/s | 18 kB 00:00 2023-10-31T12:26:41.241 INFO:teuthology.orchestra.run.smithi061.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2023-10-31T12:26:41.257 INFO:teuthology.orchestra.run.smithi061.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2023-10-31T12:26:41.283 INFO:teuthology.orchestra.run.smithi061.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.9 MB/s | 211 kB 00:00 2023-10-31T12:26:41.308 INFO:teuthology.orchestra.run.smithi061.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 856 kB/s | 43 kB 00:00 2023-10-31T12:26:41.333 INFO:teuthology.orchestra.run.smithi061.stdout:(7/13): ceph-mgr-rook-18.0.0-6792.ge83bf482.el9 168 kB/s | 48 kB 00:00 2023-10-31T12:26:41.392 INFO:teuthology.orchestra.run.smithi061.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 179 kB/s | 15 kB 00:00 2023-10-31T12:26:41.417 INFO:teuthology.orchestra.run.smithi061.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 240 kB/s | 32 kB 00:00 2023-10-31T12:26:41.476 INFO:teuthology.orchestra.run.smithi061.stdout:(10/13): python3-google-auth-2.23.3-1.el9.noarc 1.4 MB/s | 197 kB 00:00 2023-10-31T12:26:41.501 INFO:teuthology.orchestra.run.smithi061.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 706 kB/s | 59 kB 00:00 2023-10-31T12:26:41.518 INFO:teuthology.orchestra.run.smithi061.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 2.1 MB/s | 90 kB 00:00 2023-10-31T12:26:41.551 INFO:teuthology.orchestra.run.smithi061.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.4 MB/s | 1.0 MB 00:00 2023-10-31T12:26:41.553 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:26:41.554 INFO:teuthology.orchestra.run.smithi061.stdout:Total 3.7 MB/s | 2.0 MB 00:00 2023-10-31T12:26:41.554 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-10-31T12:26:41.568 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-10-31T12:26:41.568 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-10-31T12:26:41.676 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-10-31T12:26:41.676 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-10-31T12:26:42.269 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-10-31T12:26:42.439 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-10-31T12:26:42.495 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-10-31T12:26:42.551 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-10-31T12:26:42.595 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-10-31T12:26:42.637 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-10-31T12:26:42.734 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-10-31T12:26:42.852 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-google-auth-1:2.23.3-1.el9.noarch 7/13 2023-10-31T12:26:42.906 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-10-31T12:26:43.671 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-10-31T12:26:43.727 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-10-31T12:26:43.761 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-10-31T12:26:43.807 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-10-31T12:26:43.830 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mgr-rook-2:18.0.0-6792.ge83bf482.el9.noarch 13/13 2023-10-31T12:26:44.620 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6792.ge83bf482.el9.noarch 13/13 2023-10-31T12:26:44.620 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6792.ge83bf482.el9.noarch 1/13 2023-10-31T12:26:44.621 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-10-31T12:26:44.621 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-10-31T12:26:44.622 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-10-31T12:26:44.622 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-10-31T12:26:44.622 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-10-31T12:26:44.623 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-10-31T12:26:44.623 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-10-31T12:26:44.623 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-10-31T12:26:44.624 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-google-auth-1:2.23.3-1.el9.noarch 10/13 2023-10-31T12:26:44.624 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-10-31T12:26:44.624 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-10-31T12:26:45.006 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-10-31T12:26:45.007 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:45.007 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-10-31T12:26:45.007 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-rook-2:18.0.0-6792.ge83bf482.el9.noarch 2023-10-31T12:26:45.008 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-31T12:26:45.008 INFO:teuthology.orchestra.run.smithi061.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-31T12:26:45.008 INFO:teuthology.orchestra.run.smithi061.stdout: python3-google-auth-1:2.23.3-1.el9.noarch 2023-10-31T12:26:45.008 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-31T12:26:45.008 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-31T12:26:45.009 INFO:teuthology.orchestra.run.smithi061.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-31T12:26:45.009 INFO:teuthology.orchestra.run.smithi061.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-31T12:26:45.009 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-31T12:26:45.009 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-31T12:26:45.009 INFO:teuthology.orchestra.run.smithi061.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-31T12:26:45.010 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-31T12:26:45.010 INFO:teuthology.orchestra.run.smithi061.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-31T12:26:45.010 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:45.010 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:26:45.256 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-mgr-cephadm 2023-10-31T12:26:45.847 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:14 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:26:45.995 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:26:45.996 INFO:teuthology.orchestra.run.smithi061.stdout:======================================================================================== 2023-10-31T12:26:45.996 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-10-31T12:26:45.996 INFO:teuthology.orchestra.run.smithi061.stdout:======================================================================================== 2023-10-31T12:26:45.997 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-10-31T12:26:45.997 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6792.ge83bf482.el9 ceph-noarch 132 k 2023-10-31T12:26:45.997 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-10-31T12:26:45.997 INFO:teuthology.orchestra.run.smithi061.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-31T12:26:45.997 INFO:teuthology.orchestra.run.smithi061.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-31T12:26:45.998 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-31T12:26:45.998 INFO:teuthology.orchestra.run.smithi061.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-31T12:26:45.998 INFO:teuthology.orchestra.run.smithi061.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-31T12:26:45.998 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:45.998 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-10-31T12:26:45.999 INFO:teuthology.orchestra.run.smithi061.stdout:======================================================================================== 2023-10-31T12:26:45.999 INFO:teuthology.orchestra.run.smithi061.stdout:Install 6 Packages 2023-10-31T12:26:45.999 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:46.000 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 6.8 M 2023-10-31T12:26:46.000 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 33 M 2023-10-31T12:26:46.000 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-10-31T12:26:46.219 INFO:teuthology.orchestra.run.smithi061.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.6 MB/s | 228 kB 00:00 2023-10-31T12:26:46.344 INFO:teuthology.orchestra.run.smithi061.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2023-10-31T12:26:46.569 INFO:teuthology.orchestra.run.smithi061.stdout:(3/6): ceph-mgr-cephadm-18.0.0-6792.ge83bf482.e 270 kB/s | 132 kB 00:00 2023-10-31T12:26:46.911 INFO:teuthology.orchestra.run.smithi061.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 106 kB/s | 58 kB 00:00 2023-10-31T12:26:47.112 INFO:teuthology.orchestra.run.smithi061.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 94 kB/s | 51 kB 00:00 2023-10-31T12:26:47.453 INFO:teuthology.orchestra.run.smithi061.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 444 kB/s | 548 kB 00:01 2023-10-31T12:26:47.457 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:26:47.458 INFO:teuthology.orchestra.run.smithi061.stdout:Total 4.6 MB/s | 6.8 MB 00:01 2023-10-31T12:26:47.458 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-10-31T12:26:47.469 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-10-31T12:26:47.469 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-10-31T12:26:47.568 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-10-31T12:26:47.568 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-10-31T12:26:47.875 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-10-31T12:26:48.060 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-10-31T12:26:48.123 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-10-31T12:26:48.679 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-10-31T12:26:48.760 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-10-31T12:26:48.829 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-10-31T12:26:48.859 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6792.ge83bf482.el9.noarch 6/6 2023-10-31T12:26:49.660 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6792.ge83bf482.el9.noarch 6/6 2023-10-31T12:26:49.660 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6792.ge83bf482.el9.noarch 1/6 2023-10-31T12:26:49.661 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-10-31T12:26:49.661 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-10-31T12:26:49.661 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-10-31T12:26:49.661 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-10-31T12:26:49.979 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-10-31T12:26:49.979 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:49.980 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-10-31T12:26:49.980 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-cephadm-2:18.0.0-6792.ge83bf482.el9.noarch 2023-10-31T12:26:49.980 INFO:teuthology.orchestra.run.smithi061.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-31T12:26:49.980 INFO:teuthology.orchestra.run.smithi061.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-31T12:26:49.980 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-31T12:26:49.981 INFO:teuthology.orchestra.run.smithi061.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-31T12:26:49.981 INFO:teuthology.orchestra.run.smithi061.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-31T12:26:49.981 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:49.981 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:26:50.189 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-fuse 2023-10-31T12:26:50.761 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:19 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:26:50.910 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:26:50.911 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:26:50.911 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-10-31T12:26:50.911 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:26:50.911 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-10-31T12:26:50.912 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-fuse x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 833 k 2023-10-31T12:26:50.912 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-10-31T12:26:50.912 INFO:teuthology.orchestra.run.smithi061.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-31T12:26:50.912 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:50.912 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-10-31T12:26:50.913 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:26:50.913 INFO:teuthology.orchestra.run.smithi061.stdout:Install 2 Packages 2023-10-31T12:26:50.913 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:50.913 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 911 k 2023-10-31T12:26:50.913 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 3.0 M 2023-10-31T12:26:50.914 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-10-31T12:26:51.038 INFO:teuthology.orchestra.run.smithi061.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 619 kB/s | 79 kB 00:00 2023-10-31T12:26:52.088 INFO:teuthology.orchestra.run.smithi061.stdout:(2/2): ceph-fuse-18.0.0-6792.ge83bf482.el9.x86_ 707 kB/s | 833 kB 00:01 2023-10-31T12:26:52.089 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:26:52.090 INFO:teuthology.orchestra.run.smithi061.stdout:Total 772 kB/s | 911 kB 00:01 2023-10-31T12:26:52.090 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-10-31T12:26:52.116 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-10-31T12:26:52.116 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-10-31T12:26:52.201 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-10-31T12:26:52.201 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-10-31T12:26:52.395 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-10-31T12:26:52.538 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-10-31T12:26:52.616 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-fuse-2:18.0.0-6792.ge83bf482.el9.x86_64 2/2 2023-10-31T12:26:53.305 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6792.ge83bf482.el9.x86_64 2/2 2023-10-31T12:26:53.306 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-fuse-2:18.0.0-6792.ge83bf482.el9.x86_64 1/2 2023-10-31T12:26:53.651 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-10-31T12:26:53.652 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:53.652 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-10-31T12:26:53.652 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-fuse-2:18.0.0-6792.ge83bf482.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-10-31T12:26:53.653 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:53.653 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:26:53.932 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install ceph-volume 2023-10-31T12:26:54.506 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:23 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:26:54.656 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:26:54.657 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:26:54.657 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-10-31T12:26:54.657 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:26:54.657 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-10-31T12:26:54.657 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-volume noarch 2:18.0.0-6792.ge83bf482.el9 ceph-noarch 262 k 2023-10-31T12:26:54.658 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:54.658 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-10-31T12:26:54.658 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:26:54.658 INFO:teuthology.orchestra.run.smithi061.stdout:Install 1 Package 2023-10-31T12:26:54.658 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:54.659 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 262 k 2023-10-31T12:26:54.659 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 1.2 M 2023-10-31T12:26:54.659 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-10-31T12:26:54.944 INFO:teuthology.orchestra.run.smithi061.stdout:ceph-volume-18.0.0-6792.ge83bf482.el9.noarch.rp 913 kB/s | 262 kB 00:00 2023-10-31T12:26:54.945 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:26:54.945 INFO:teuthology.orchestra.run.smithi061.stdout:Total 909 kB/s | 262 kB 00:00 2023-10-31T12:26:54.945 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-10-31T12:26:54.954 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-10-31T12:26:54.955 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-10-31T12:26:54.995 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-10-31T12:26:54.995 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-10-31T12:26:55.222 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-10-31T12:26:55.288 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : ceph-volume-2:18.0.0-6792.ge83bf482.el9.noarch 1/1 2023-10-31T12:26:55.304 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: ceph-volume-2:18.0.0-6792.ge83bf482.el9.noarch 1/1 2023-10-31T12:26:55.305 INFO:teuthology.orchestra.run.smithi061.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-31T12:26:55.305 INFO:teuthology.orchestra.run.smithi061.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-31T12:26:55.305 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:56.353 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : ceph-volume-2:18.0.0-6792.ge83bf482.el9.noarch 1/1 2023-10-31T12:26:56.354 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:56.354 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-10-31T12:26:56.354 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-volume-2:18.0.0-6792.ge83bf482.el9.noarch 2023-10-31T12:26:56.355 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:56.355 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:26:56.635 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install librados-devel 2023-10-31T12:26:57.215 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:26 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:26:57.364 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:26:57.365 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:26:57.365 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repo Size 2023-10-31T12:26:57.365 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:26:57.365 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-10-31T12:26:57.365 INFO:teuthology.orchestra.run.smithi061.stdout: librados-devel x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 127 k 2023-10-31T12:26:57.366 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:57.366 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-10-31T12:26:57.366 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:26:57.366 INFO:teuthology.orchestra.run.smithi061.stdout:Install 1 Package 2023-10-31T12:26:57.367 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:57.367 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 127 k 2023-10-31T12:26:57.367 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 456 k 2023-10-31T12:26:57.367 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-10-31T12:26:57.807 INFO:teuthology.orchestra.run.smithi061.stdout:librados-devel-18.0.0-6792.ge83bf482.el9.x86_64 287 kB/s | 127 kB 00:00 2023-10-31T12:26:57.808 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:26:57.809 INFO:teuthology.orchestra.run.smithi061.stdout:Total 286 kB/s | 127 kB 00:00 2023-10-31T12:26:57.809 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-10-31T12:26:57.814 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-10-31T12:26:57.815 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-10-31T12:26:57.874 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-10-31T12:26:57.874 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-10-31T12:26:58.020 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-10-31T12:26:58.150 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : librados-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:26:58.619 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: librados-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:26:58.887 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librados-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:26:58.887 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:58.887 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-10-31T12:26:58.888 INFO:teuthology.orchestra.run.smithi061.stdout: librados-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:26:58.888 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:26:58.888 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:26:59.118 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install libcephfs2 2023-10-31T12:26:59.693 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:28 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:26:59.790 INFO:teuthology.orchestra.run.smithi061.stdout:Package libcephfs2-2:18.0.0-6792.ge83bf482.el9.x86_64 is already installed. 2023-10-31T12:26:59.842 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:26:59.843 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-10-31T12:26:59.843 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:26:59.907 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install libcephfs-devel 2023-10-31T12:27:00.480 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:29 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:27:00.629 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:27:00.630 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:27:00.630 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repo Size 2023-10-31T12:27:00.631 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:27:00.631 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-10-31T12:27:00.631 INFO:teuthology.orchestra.run.smithi061.stdout: libcephfs-devel x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 31 k 2023-10-31T12:27:00.631 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:00.631 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-10-31T12:27:00.632 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:27:00.632 INFO:teuthology.orchestra.run.smithi061.stdout:Install 1 Package 2023-10-31T12:27:00.632 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:00.632 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 31 k 2023-10-31T12:27:00.632 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 139 k 2023-10-31T12:27:00.633 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-10-31T12:27:00.844 INFO:teuthology.orchestra.run.smithi061.stdout:libcephfs-devel-18.0.0-6792.ge83bf482.el9.x86_6 146 kB/s | 31 kB 00:00 2023-10-31T12:27:00.845 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:27:00.845 INFO:teuthology.orchestra.run.smithi061.stdout:Total 145 kB/s | 31 kB 00:00 2023-10-31T12:27:00.845 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-10-31T12:27:00.849 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-10-31T12:27:00.849 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-10-31T12:27:00.885 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-10-31T12:27:00.885 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-10-31T12:27:00.975 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-10-31T12:27:01.095 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libcephfs-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:27:01.492 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:27:01.770 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libcephfs-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:27:01.770 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:01.771 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-10-31T12:27:01.771 INFO:teuthology.orchestra.run.smithi061.stdout: libcephfs-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:27:01.771 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:01.771 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:27:01.950 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install librados2 2023-10-31T12:27:02.519 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:31 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:27:02.615 INFO:teuthology.orchestra.run.smithi061.stdout:Package librados2-2:18.0.0-6792.ge83bf482.el9.x86_64 is already installed. 2023-10-31T12:27:02.668 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:27:02.669 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-10-31T12:27:02.669 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:27:02.733 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install librbd1 2023-10-31T12:27:03.301 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:32 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:27:03.400 INFO:teuthology.orchestra.run.smithi061.stdout:Package librbd1-2:18.0.0-6792.ge83bf482.el9.x86_64 is already installed. 2023-10-31T12:27:03.454 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:27:03.514 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-10-31T12:27:03.514 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:27:03.516 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install python3-rados 2023-10-31T12:27:04.085 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:33 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:27:04.180 INFO:teuthology.orchestra.run.smithi061.stdout:Package python3-rados-2:18.0.0-6792.ge83bf482.el9.x86_64 is already installed. 2023-10-31T12:27:04.232 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:27:04.233 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-10-31T12:27:04.233 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:27:04.299 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install python3-rgw 2023-10-31T12:27:04.863 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:33 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:27:04.958 INFO:teuthology.orchestra.run.smithi061.stdout:Package python3-rgw-2:18.0.0-6792.ge83bf482.el9.x86_64 is already installed. 2023-10-31T12:27:05.013 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:27:05.014 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-10-31T12:27:05.014 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:27:05.076 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install python3-cephfs 2023-10-31T12:27:05.649 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:34 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:27:05.745 INFO:teuthology.orchestra.run.smithi061.stdout:Package python3-cephfs-2:18.0.0-6792.ge83bf482.el9.x86_64 is already installed. 2023-10-31T12:27:05.797 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:27:05.798 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-10-31T12:27:05.799 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:27:05.858 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install python3-rbd 2023-10-31T12:27:06.426 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:35 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:27:06.521 INFO:teuthology.orchestra.run.smithi061.stdout:Package python3-rbd-2:18.0.0-6792.ge83bf482.el9.x86_64 is already installed. 2023-10-31T12:27:06.574 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:27:06.575 INFO:teuthology.orchestra.run.smithi061.stdout:Nothing to do. 2023-10-31T12:27:06.575 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:27:06.637 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install rbd-fuse 2023-10-31T12:27:07.208 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:36 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:27:07.357 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:27:07.358 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:27:07.358 INFO:teuthology.orchestra.run.smithi061.stdout: Package Architecture Version Repository Size 2023-10-31T12:27:07.358 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:27:07.359 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-10-31T12:27:07.359 INFO:teuthology.orchestra.run.smithi061.stdout: rbd-fuse x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 87 k 2023-10-31T12:27:07.359 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:07.359 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-10-31T12:27:07.359 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:27:07.360 INFO:teuthology.orchestra.run.smithi061.stdout:Install 1 Package 2023-10-31T12:27:07.360 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:07.360 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 87 k 2023-10-31T12:27:07.360 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 252 k 2023-10-31T12:27:07.360 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-10-31T12:27:07.631 INFO:teuthology.orchestra.run.smithi061.stdout:rbd-fuse-18.0.0-6792.ge83bf482.el9.x86_64.rpm 319 kB/s | 87 kB 00:00 2023-10-31T12:27:07.631 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:27:07.632 INFO:teuthology.orchestra.run.smithi061.stdout:Total 318 kB/s | 87 kB 00:00 2023-10-31T12:27:07.632 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-10-31T12:27:07.639 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-10-31T12:27:07.639 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-10-31T12:27:07.698 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-10-31T12:27:07.699 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-10-31T12:27:07.844 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-10-31T12:27:07.949 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : rbd-fuse-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:27:08.315 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:27:08.661 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : rbd-fuse-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:27:08.662 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:08.662 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-10-31T12:27:08.662 INFO:teuthology.orchestra.run.smithi061.stdout: rbd-fuse-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:27:08.662 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:08.663 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:27:08.884 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install rbd-mirror 2023-10-31T12:27:09.474 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:38 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:27:09.625 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:27:09.626 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:27:09.626 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repo Size 2023-10-31T12:27:09.626 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:27:09.626 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-10-31T12:27:09.627 INFO:teuthology.orchestra.run.smithi061.stdout: rbd-mirror x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 3.0 M 2023-10-31T12:27:09.627 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:09.627 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-10-31T12:27:09.627 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:27:09.627 INFO:teuthology.orchestra.run.smithi061.stdout:Install 1 Package 2023-10-31T12:27:09.628 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:09.628 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 3.0 M 2023-10-31T12:27:09.628 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 13 M 2023-10-31T12:27:09.628 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-10-31T12:27:11.076 INFO:teuthology.orchestra.run.smithi061.stdout:rbd-mirror-18.0.0-6792.ge83bf482.el9.x86_64.rpm 2.1 MB/s | 3.0 MB 00:01 2023-10-31T12:27:11.077 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:27:11.077 INFO:teuthology.orchestra.run.smithi061.stdout:Total 2.1 MB/s | 3.0 MB 00:01 2023-10-31T12:27:11.078 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-10-31T12:27:11.089 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-10-31T12:27:11.090 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-10-31T12:27:11.170 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-10-31T12:27:11.171 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-10-31T12:27:11.617 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-10-31T12:27:11.668 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : rbd-mirror-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:27:11.694 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:27:11.695 INFO:teuthology.orchestra.run.smithi061.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-31T12:27:11.695 INFO:teuthology.orchestra.run.smithi061.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-31T12:27:11.695 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-31T12:27:11.695 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-31T12:27:11.695 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:12.910 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : rbd-mirror-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:27:12.911 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:12.911 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-10-31T12:27:12.911 INFO:teuthology.orchestra.run.smithi061.stdout: rbd-mirror-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:27:12.911 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:12.912 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:27:13.158 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install rbd-nbd 2023-10-31T12:27:13.737 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:02:42 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:27:13.885 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:27:13.886 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:27:13.886 INFO:teuthology.orchestra.run.smithi061.stdout: Package Architecture Version Repository Size 2023-10-31T12:27:13.886 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:27:13.887 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-10-31T12:27:13.887 INFO:teuthology.orchestra.run.smithi061.stdout: rbd-nbd x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 168 k 2023-10-31T12:27:13.887 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:13.887 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-10-31T12:27:13.887 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:27:13.888 INFO:teuthology.orchestra.run.smithi061.stdout:Install 1 Package 2023-10-31T12:27:13.888 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:13.888 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 168 k 2023-10-31T12:27:13.888 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 532 k 2023-10-31T12:27:13.888 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-10-31T12:27:14.103 INFO:teuthology.orchestra.run.smithi061.stdout:rbd-nbd-18.0.0-6792.ge83bf482.el9.x86_64.rpm 779 kB/s | 168 kB 00:00 2023-10-31T12:27:14.104 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:27:14.104 INFO:teuthology.orchestra.run.smithi061.stdout:Total 773 kB/s | 168 kB 00:00 2023-10-31T12:27:14.105 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-10-31T12:27:14.113 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-10-31T12:27:14.113 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-10-31T12:27:14.172 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-10-31T12:27:14.172 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-10-31T12:27:14.322 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-10-31T12:27:14.413 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : rbd-nbd-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:27:14.822 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:27:15.109 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : rbd-nbd-2:18.0.0-6792.ge83bf482.el9.x86_64 1/1 2023-10-31T12:27:15.110 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:15.110 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-10-31T12:27:15.110 INFO:teuthology.orchestra.run.smithi061.stdout: rbd-nbd-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:27:15.111 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:15.111 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:27:15.338 DEBUG:teuthology.parallel:result is None 2023-10-31T12:27:15.338 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=e83bf4821bfb604694b03a67d7390bfb38655506 2023-10-31T12:27:15.483 DEBUG:teuthology.orchestra.run.smithi061:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-31T12:27:15.504 INFO:teuthology.orchestra.run.smithi061.stdout:18.0.0-6792.ge83bf482.el9 2023-10-31T12:27:15.504 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6792.ge83bf482.el9 2023-10-31T12:27:15.505 INFO:teuthology.task.install:The correct ceph version 18.0.0-6792.ge83bf482 is installed. 2023-10-31T12:27:15.505 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-31T12:27:15.506 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:27:15.506 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-31T12:27:15.575 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-31T12:27:15.576 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:27:15.576 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/usr/bin/daemon-helper 2023-10-31T12:27:15.645 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-31T12:27:15.715 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-31T12:27:15.715 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:27:15.716 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-31T12:27:15.785 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-31T12:27:15.855 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-31T12:27:15.856 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:27:15.856 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/usr/bin/stdin-killer 2023-10-31T12:27:15.925 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-31T12:27:15.995 INFO:teuthology.run_tasks:Running task ceph... 2023-10-31T12:27:16.059 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-31T12:27:16.060 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 777 /var/log/ceph 2023-10-31T12:27:16.091 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-31T12:27:16.091 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-31T12:27:16.161 INFO:tasks.ceph:Creating extra log directories... 2023-10-31T12:27:16.162 DEBUG:teuthology.orchestra.run.smithi061:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-31T12:27:16.239 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-31T12:27:16.239 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-31T12:27:16.240 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7441974', 'branch': 'wip-wip-matanb-crimson-do_osd_ops_execute-v3', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_seq_read}', 'email': None, 'first_in_suite': False, 'job_id': '7441974', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-wip-matanb-crimson-do_osd_ops_execute-v3'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'e83bf4821bfb604694b03a67d7390bfb38655506'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': 'e83bf4821bfb604694b03a67d7390bfb38655506'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '21548fe806cf259deac1421530d5ce720be17997'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 1530, 'sha1': 'e83bf4821bfb604694b03a67d7390bfb38655506', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_21548fe806cf259deac1421530d5ce720be17997/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': '21548fe806cf259deac1421530d5ce720be17997', 'targets': {'smithi061.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLrJA40Hkx/RPJJwnSiZcODOwfqW/iC7Q/PzuwzQah+m/ONKi/hvZW56fzMf8XhGcoKh5u3N8Bc1cNqBLauDHrk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'e83bf4821bfb604694b03a67d7390bfb38655506', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-10-31_11:40:33', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.770726'} 2023-10-31T12:27:16.241 DEBUG:teuthology.orchestra.run.smithi061:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-31T12:27:16.286 DEBUG:teuthology.orchestra.run.smithi061:> sudo install -d -m0777 -- /var/run/ceph 2023-10-31T12:27:16.356 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:27:16.357 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/scratch_devs of=/dev/stdout 2023-10-31T12:27:16.412 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-31T12:27:16.413 DEBUG:teuthology.orchestra.run.smithi061:> stat /dev/vg_nvme/lv_1 2023-10-31T12:27:16.467 INFO:teuthology.orchestra.run.smithi061.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-31T12:27:16.467 INFO:teuthology.orchestra.run.smithi061.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-31T12:27:16.468 INFO:teuthology.orchestra.run.smithi061.stdout:Device: 5h/5d Inode: 662 Links: 1 2023-10-31T12:27:16.468 INFO:teuthology.orchestra.run.smithi061.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-31T12:27:16.469 INFO:teuthology.orchestra.run.smithi061.stdout:Context: system_u:object_r:device_t:s0 2023-10-31T12:27:16.469 INFO:teuthology.orchestra.run.smithi061.stdout:Access: 2023-10-31 12:27:11.939005075 +0000 2023-10-31T12:27:16.469 INFO:teuthology.orchestra.run.smithi061.stdout:Modify: 2023-10-31 12:23:28.690831820 +0000 2023-10-31T12:27:16.470 INFO:teuthology.orchestra.run.smithi061.stdout:Change: 2023-10-31 12:23:28.690831820 +0000 2023-10-31T12:27:16.470 INFO:teuthology.orchestra.run.smithi061.stdout: Birth: 2023-10-31 12:23:28.690831820 +0000 2023-10-31T12:27:16.470 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-31T12:27:16.536 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records in 2023-10-31T12:27:16.536 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records out 2023-10-31T12:27:16.536 INFO:teuthology.orchestra.run.smithi061.stderr:512 bytes copied, 0.000179769 s, 2.8 MB/s 2023-10-31T12:27:16.537 DEBUG:teuthology.orchestra.run.smithi061:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-31T12:27:16.593 DEBUG:teuthology.orchestra.run.smithi061:> stat /dev/vg_nvme/lv_2 2023-10-31T12:27:16.648 INFO:teuthology.orchestra.run.smithi061.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-31T12:27:16.649 INFO:teuthology.orchestra.run.smithi061.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-31T12:27:16.649 INFO:teuthology.orchestra.run.smithi061.stdout:Device: 5h/5d Inode: 682 Links: 1 2023-10-31T12:27:16.649 INFO:teuthology.orchestra.run.smithi061.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-31T12:27:16.649 INFO:teuthology.orchestra.run.smithi061.stdout:Context: system_u:object_r:device_t:s0 2023-10-31T12:27:16.649 INFO:teuthology.orchestra.run.smithi061.stdout:Access: 2023-10-31 12:27:11.939005075 +0000 2023-10-31T12:27:16.650 INFO:teuthology.orchestra.run.smithi061.stdout:Modify: 2023-10-31 12:23:29.206824037 +0000 2023-10-31T12:27:16.650 INFO:teuthology.orchestra.run.smithi061.stdout:Change: 2023-10-31 12:23:29.206824037 +0000 2023-10-31T12:27:16.650 INFO:teuthology.orchestra.run.smithi061.stdout: Birth: 2023-10-31 12:23:29.206824037 +0000 2023-10-31T12:27:16.650 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-31T12:27:16.717 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records in 2023-10-31T12:27:16.717 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records out 2023-10-31T12:27:16.717 INFO:teuthology.orchestra.run.smithi061.stderr:512 bytes copied, 0.000193838 s, 2.6 MB/s 2023-10-31T12:27:16.718 DEBUG:teuthology.orchestra.run.smithi061:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-31T12:27:16.775 DEBUG:teuthology.orchestra.run.smithi061:> stat /dev/vg_nvme/lv_3 2023-10-31T12:27:16.830 INFO:teuthology.orchestra.run.smithi061.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-31T12:27:16.830 INFO:teuthology.orchestra.run.smithi061.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-31T12:27:16.831 INFO:teuthology.orchestra.run.smithi061.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-10-31T12:27:16.831 INFO:teuthology.orchestra.run.smithi061.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-31T12:27:16.831 INFO:teuthology.orchestra.run.smithi061.stdout:Context: system_u:object_r:device_t:s0 2023-10-31T12:27:16.831 INFO:teuthology.orchestra.run.smithi061.stdout:Access: 2023-10-31 12:27:11.940005060 +0000 2023-10-31T12:27:16.831 INFO:teuthology.orchestra.run.smithi061.stdout:Modify: 2023-10-31 12:23:29.702816556 +0000 2023-10-31T12:27:16.831 INFO:teuthology.orchestra.run.smithi061.stdout:Change: 2023-10-31 12:23:29.702816556 +0000 2023-10-31T12:27:16.832 INFO:teuthology.orchestra.run.smithi061.stdout: Birth: 2023-10-31 12:23:29.702816556 +0000 2023-10-31T12:27:16.832 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-31T12:27:16.899 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records in 2023-10-31T12:27:16.899 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records out 2023-10-31T12:27:16.900 INFO:teuthology.orchestra.run.smithi061.stderr:512 bytes copied, 0.000167148 s, 3.1 MB/s 2023-10-31T12:27:16.901 DEBUG:teuthology.orchestra.run.smithi061:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-31T12:27:16.957 DEBUG:teuthology.orchestra.run.smithi061:> stat /dev/vg_nvme/lv_4 2023-10-31T12:27:17.012 INFO:teuthology.orchestra.run.smithi061.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-31T12:27:17.012 INFO:teuthology.orchestra.run.smithi061.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-31T12:27:17.012 INFO:teuthology.orchestra.run.smithi061.stdout:Device: 5h/5d Inode: 721 Links: 1 2023-10-31T12:27:17.013 INFO:teuthology.orchestra.run.smithi061.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-31T12:27:17.013 INFO:teuthology.orchestra.run.smithi061.stdout:Context: system_u:object_r:device_t:s0 2023-10-31T12:27:17.013 INFO:teuthology.orchestra.run.smithi061.stdout:Access: 2023-10-31 12:27:11.940005060 +0000 2023-10-31T12:27:17.013 INFO:teuthology.orchestra.run.smithi061.stdout:Modify: 2023-10-31 12:23:30.204808985 +0000 2023-10-31T12:27:17.013 INFO:teuthology.orchestra.run.smithi061.stdout:Change: 2023-10-31 12:23:30.204808985 +0000 2023-10-31T12:27:17.014 INFO:teuthology.orchestra.run.smithi061.stdout: Birth: 2023-10-31 12:23:30.204808985 +0000 2023-10-31T12:27:17.014 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-31T12:27:17.081 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records in 2023-10-31T12:27:17.081 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records out 2023-10-31T12:27:17.082 INFO:teuthology.orchestra.run.smithi061.stderr:512 bytes copied, 0.000178743 s, 2.9 MB/s 2023-10-31T12:27:17.083 DEBUG:teuthology.orchestra.run.smithi061:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-31T12:27:17.139 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-31T12:27:17.139 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi061.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-31T12:27:17.139 INFO:tasks.ceph:Generating config... 2023-10-31T12:27:17.141 INFO:tasks.ceph:[osd] debug monc = 20 2023-10-31T12:27:17.141 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-31T12:27:17.141 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2023-10-31T12:27:17.141 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-10-31T12:27:17.142 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-31T12:27:17.142 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2023-10-31T12:27:17.142 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2023-10-31T12:27:17.143 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-10-31T12:27:17.143 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-10-31T12:27:17.143 INFO:tasks.ceph:[osd] debug filestore = 0/0 2023-10-31T12:27:17.143 INFO:tasks.ceph:[osd] debug journal = 0/0 2023-10-31T12:27:17.144 INFO:tasks.ceph:[osd] debug ms = 0/0 2023-10-31T12:27:17.144 INFO:tasks.ceph:[osd] debug osd = 0/0 2023-10-31T12:27:17.144 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-10-31T12:27:17.145 INFO:tasks.ceph:[osd] err to stderr = True 2023-10-31T12:27:17.145 INFO:tasks.ceph:[osd] log flush on exit = True 2023-10-31T12:27:17.145 INFO:tasks.ceph:[osd] log to file = False 2023-10-31T12:27:17.145 INFO:tasks.ceph:[osd] log to stderr = True 2023-10-31T12:27:17.146 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-31T12:27:17.146 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-31T12:27:17.146 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-31T12:27:17.147 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-31T12:27:17.147 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-31T12:27:17.147 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-31T12:27:17.147 INFO:tasks.ceph:[global] auth client required = none 2023-10-31T12:27:17.148 INFO:tasks.ceph:[global] auth cluster required = none 2023-10-31T12:27:17.148 INFO:tasks.ceph:[global] auth service required = none 2023-10-31T12:27:17.148 INFO:tasks.ceph:[global] auth supported = none 2023-10-31T12:27:17.148 INFO:tasks.ceph:[global] debug asok = 0/0 2023-10-31T12:27:17.149 INFO:tasks.ceph:[global] debug auth = 0/0 2023-10-31T12:27:17.149 INFO:tasks.ceph:[global] debug bdev = 0/0 2023-10-31T12:27:17.149 INFO:tasks.ceph:[global] debug bluefs = 0/0 2023-10-31T12:27:17.149 INFO:tasks.ceph:[global] debug bluestore = 0/0 2023-10-31T12:27:17.150 INFO:tasks.ceph:[global] debug buffer = 0/0 2023-10-31T12:27:17.150 INFO:tasks.ceph:[global] debug civetweb = 0/0 2023-10-31T12:27:17.150 INFO:tasks.ceph:[global] debug client = 0/0 2023-10-31T12:27:17.151 INFO:tasks.ceph:[global] debug compressor = 0/0 2023-10-31T12:27:17.151 INFO:tasks.ceph:[global] debug context = 0/0 2023-10-31T12:27:17.151 INFO:tasks.ceph:[global] debug crush = 0/0 2023-10-31T12:27:17.151 INFO:tasks.ceph:[global] debug crypto = 0/0 2023-10-31T12:27:17.152 INFO:tasks.ceph:[global] debug dpdk = 0/0 2023-10-31T12:27:17.152 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2023-10-31T12:27:17.152 INFO:tasks.ceph:[global] debug filer = 0/0 2023-10-31T12:27:17.152 INFO:tasks.ceph:[global] debug filestore = 0/0 2023-10-31T12:27:17.153 INFO:tasks.ceph:[global] debug finisher = 0/0 2023-10-31T12:27:17.153 INFO:tasks.ceph:[global] debug fuse = 0/0 2023-10-31T12:27:17.153 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2023-10-31T12:27:17.154 INFO:tasks.ceph:[global] debug javaclient = 0/0 2023-10-31T12:27:17.154 INFO:tasks.ceph:[global] debug journal = 0/0 2023-10-31T12:27:17.154 INFO:tasks.ceph:[global] debug journaler = 0/0 2023-10-31T12:27:17.154 INFO:tasks.ceph:[global] debug kstore = 0/0 2023-10-31T12:27:17.155 INFO:tasks.ceph:[global] debug leveldb = 0/0 2023-10-31T12:27:17.155 INFO:tasks.ceph:[global] debug lockdep = 0/0 2023-10-31T12:27:17.155 INFO:tasks.ceph:[global] debug mds = 0/0 2023-10-31T12:27:17.155 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2023-10-31T12:27:17.156 INFO:tasks.ceph:[global] debug mds locker = 0/0 2023-10-31T12:27:17.156 INFO:tasks.ceph:[global] debug mds log = 0/0 2023-10-31T12:27:17.156 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2023-10-31T12:27:17.156 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2023-10-31T12:27:17.157 INFO:tasks.ceph:[global] debug memdb = 0/0 2023-10-31T12:27:17.157 INFO:tasks.ceph:[global] debug mgr = 0/0 2023-10-31T12:27:17.157 INFO:tasks.ceph:[global] debug mgrc = 0/0 2023-10-31T12:27:17.157 INFO:tasks.ceph:[global] debug mon = 0/0 2023-10-31T12:27:17.158 INFO:tasks.ceph:[global] debug monc = 0/0 2023-10-31T12:27:17.158 INFO:tasks.ceph:[global] debug ms = 0/0 2023-10-31T12:27:17.158 INFO:tasks.ceph:[global] debug objclass = 0/0 2023-10-31T12:27:17.158 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2023-10-31T12:27:17.159 INFO:tasks.ceph:[global] debug objecter = 0/0 2023-10-31T12:27:17.159 INFO:tasks.ceph:[global] debug optracker = 0/0 2023-10-31T12:27:17.159 INFO:tasks.ceph:[global] debug osd = 0/0 2023-10-31T12:27:17.159 INFO:tasks.ceph:[global] debug paxos = 0/0 2023-10-31T12:27:17.160 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2023-10-31T12:27:17.160 INFO:tasks.ceph:[global] debug rados = 0/0 2023-10-31T12:27:17.160 INFO:tasks.ceph:[global] debug rbd = 0/0 2023-10-31T12:27:17.160 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2023-10-31T12:27:17.161 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2023-10-31T12:27:17.161 INFO:tasks.ceph:[global] debug refs = 0/0 2023-10-31T12:27:17.161 INFO:tasks.ceph:[global] debug rgw = 0/0 2023-10-31T12:27:17.161 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2023-10-31T12:27:17.162 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2023-10-31T12:27:17.162 INFO:tasks.ceph:[global] debug striper = 0/0 2023-10-31T12:27:17.162 INFO:tasks.ceph:[global] debug throttle = 0/0 2023-10-31T12:27:17.163 INFO:tasks.ceph:[global] debug timer = 0/0 2023-10-31T12:27:17.163 INFO:tasks.ceph:[global] debug tp = 0/0 2023-10-31T12:27:17.163 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2023-10-31T12:27:17.163 INFO:tasks.ceph:[global] ms client mode = crc 2023-10-31T12:27:17.164 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-10-31T12:27:17.164 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-10-31T12:27:17.164 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-10-31T12:27:17.164 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-10-31T12:27:17.165 INFO:tasks.ceph:[global] ms service mode = crc 2023-10-31T12:27:17.165 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-31T12:27:17.165 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-31T12:27:17.165 INFO:tasks.ceph:[mon] debug mon = 0/0 2023-10-31T12:27:17.166 INFO:tasks.ceph:[mon] debug ms = 0/0 2023-10-31T12:27:17.166 INFO:tasks.ceph:[mon] debug paxos = 0/0 2023-10-31T12:27:17.166 INFO:tasks.ceph:[mon] osd pool default crimson = True 2023-10-31T12:27:17.167 INFO:tasks.ceph:Setting up mon.a... 2023-10-31T12:27:17.167 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-31T12:27:17.235 INFO:teuthology.orchestra.run.smithi061.stdout:creating /etc/ceph/ceph.keyring 2023-10-31T12:27:17.239 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-31T12:27:17.291 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-31T12:27:17.322 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.61')] 2023-10-31T12:27:17.323 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.61', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-10-31T12:27:17.324 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:27:17.324 DEBUG:teuthology.orchestra.run.smithi061:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-31T12:27:17.377 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.61 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-31T12:27:17.463 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:17.460+0000 7fb424e91e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:17.464 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:17.461+0000 7fb424e91e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:17.464 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:17.461+0000 7fb424e91e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:17.465 INFO:teuthology.orchestra.run.smithi061.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-31T12:27:17.465 INFO:teuthology.orchestra.run.smithi061.stdout:monmaptool: generated fsid 30686b99-d4b7-4afe-9de4-18fff3a2aa8e 2023-10-31T12:27:17.465 INFO:teuthology.orchestra.run.smithi061.stdout:setting min_mon_release = pacific 2023-10-31T12:27:17.465 INFO:teuthology.orchestra.run.smithi061.stdout:epoch 0 2023-10-31T12:27:17.466 INFO:teuthology.orchestra.run.smithi061.stdout:fsid 30686b99-d4b7-4afe-9de4-18fff3a2aa8e 2023-10-31T12:27:17.466 INFO:teuthology.orchestra.run.smithi061.stdout:last_changed 2023-10-31T12:27:17.462710+0000 2023-10-31T12:27:17.466 INFO:teuthology.orchestra.run.smithi061.stdout:created 2023-10-31T12:27:17.462710+0000 2023-10-31T12:27:17.466 INFO:teuthology.orchestra.run.smithi061.stdout:min_mon_release 16 (pacific) 2023-10-31T12:27:17.467 INFO:teuthology.orchestra.run.smithi061.stdout:election_strategy: 1 2023-10-31T12:27:17.467 INFO:teuthology.orchestra.run.smithi061.stdout:0: [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] mon.a 2023-10-31T12:27:17.467 INFO:teuthology.orchestra.run.smithi061.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-31T12:27:17.468 DEBUG:teuthology.orchestra.run.smithi061:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-31T12:27:17.483 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 30686b99-d4b7-4afe-9de4-18fff3a2aa8e... 2023-10-31T12:27:17.484 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-31T12:27:17.583 INFO:teuthology.orchestra.run.smithi061.stdout:[global] 2023-10-31T12:27:17.583 INFO:teuthology.orchestra.run.smithi061.stdout: chdir = "" 2023-10-31T12:27:17.584 INFO:teuthology.orchestra.run.smithi061.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-31T12:27:17.584 INFO:teuthology.orchestra.run.smithi061.stdout: auth supported = none 2023-10-31T12:27:17.584 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.584 INFO:teuthology.orchestra.run.smithi061.stdout: filestore xattr use omap = true 2023-10-31T12:27:17.584 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.585 INFO:teuthology.orchestra.run.smithi061.stdout: mon clock drift allowed = 1.000 2023-10-31T12:27:17.585 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.585 INFO:teuthology.orchestra.run.smithi061.stdout: osd crush chooseleaf type = 0 2023-10-31T12:27:17.585 INFO:teuthology.orchestra.run.smithi061.stdout: auth debug = true 2023-10-31T12:27:17.585 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.586 INFO:teuthology.orchestra.run.smithi061.stdout: ms die on old message = true 2023-10-31T12:27:17.586 INFO:teuthology.orchestra.run.smithi061.stdout: ms die on bug = true 2023-10-31T12:27:17.586 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.586 INFO:teuthology.orchestra.run.smithi061.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-31T12:27:17.586 INFO:teuthology.orchestra.run.smithi061.stdout: mon pg warn max object skew = 0 2023-10-31T12:27:17.587 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.587 INFO:teuthology.orchestra.run.smithi061.stdout: # disable pg_autoscaler by default for new pools 2023-10-31T12:27:17.587 INFO:teuthology.orchestra.run.smithi061.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-31T12:27:17.587 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.587 INFO:teuthology.orchestra.run.smithi061.stdout: osd pool default size = 2 2023-10-31T12:27:17.588 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.588 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd allow primary affinity = true 2023-10-31T12:27:17.588 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd allow pg remap = true 2023-10-31T12:27:17.588 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on legacy crush tunables = false 2023-10-31T12:27:17.588 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on crush straw calc version zero = false 2023-10-31T12:27:17.589 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on no sortbitwise = false 2023-10-31T12:27:17.589 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on osd down out interval zero = false 2023-10-31T12:27:17.589 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on too few osds = false 2023-10-31T12:27:17.589 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-31T12:27:17.589 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-31T12:27:17.590 INFO:teuthology.orchestra.run.smithi061.stdout: mon_allow_pool_size_one = true 2023-10-31T12:27:17.590 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.590 INFO:teuthology.orchestra.run.smithi061.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-31T12:27:17.590 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.590 INFO:teuthology.orchestra.run.smithi061.stdout: osd default data pool replay window = 5 2023-10-31T12:27:17.591 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.591 INFO:teuthology.orchestra.run.smithi061.stdout: mon allow pool delete = true 2023-10-31T12:27:17.591 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.591 INFO:teuthology.orchestra.run.smithi061.stdout: mon cluster log file level = debug 2023-10-31T12:27:17.591 INFO:teuthology.orchestra.run.smithi061.stdout: debug asserts on shutdown = true 2023-10-31T12:27:17.591 INFO:teuthology.orchestra.run.smithi061.stdout: mon health detail to clog = false 2023-10-31T12:27:17.592 INFO:teuthology.orchestra.run.smithi061.stdout: mon host = 172.21.15.61 2023-10-31T12:27:17.592 INFO:teuthology.orchestra.run.smithi061.stdout: auth client required = none 2023-10-31T12:27:17.592 INFO:teuthology.orchestra.run.smithi061.stdout: auth cluster required = none 2023-10-31T12:27:17.592 INFO:teuthology.orchestra.run.smithi061.stdout: auth service required = none 2023-10-31T12:27:17.592 INFO:teuthology.orchestra.run.smithi061.stdout: debug asok = 0/0 2023-10-31T12:27:17.593 INFO:teuthology.orchestra.run.smithi061.stdout: debug auth = 0/0 2023-10-31T12:27:17.593 INFO:teuthology.orchestra.run.smithi061.stdout: debug bdev = 0/0 2023-10-31T12:27:17.593 INFO:teuthology.orchestra.run.smithi061.stdout: debug bluefs = 0/0 2023-10-31T12:27:17.593 INFO:teuthology.orchestra.run.smithi061.stdout: debug bluestore = 0/0 2023-10-31T12:27:17.593 INFO:teuthology.orchestra.run.smithi061.stdout: debug buffer = 0/0 2023-10-31T12:27:17.594 INFO:teuthology.orchestra.run.smithi061.stdout: debug civetweb = 0/0 2023-10-31T12:27:17.594 INFO:teuthology.orchestra.run.smithi061.stdout: debug client = 0/0 2023-10-31T12:27:17.594 INFO:teuthology.orchestra.run.smithi061.stdout: debug compressor = 0/0 2023-10-31T12:27:17.594 INFO:teuthology.orchestra.run.smithi061.stdout: debug context = 0/0 2023-10-31T12:27:17.594 INFO:teuthology.orchestra.run.smithi061.stdout: debug crush = 0/0 2023-10-31T12:27:17.595 INFO:teuthology.orchestra.run.smithi061.stdout: debug crypto = 0/0 2023-10-31T12:27:17.595 INFO:teuthology.orchestra.run.smithi061.stdout: debug dpdk = 0/0 2023-10-31T12:27:17.595 INFO:teuthology.orchestra.run.smithi061.stdout: debug eventtrace = 0/0 2023-10-31T12:27:17.595 INFO:teuthology.orchestra.run.smithi061.stdout: debug filer = 0/0 2023-10-31T12:27:17.595 INFO:teuthology.orchestra.run.smithi061.stdout: debug filestore = 0/0 2023-10-31T12:27:17.596 INFO:teuthology.orchestra.run.smithi061.stdout: debug finisher = 0/0 2023-10-31T12:27:17.596 INFO:teuthology.orchestra.run.smithi061.stdout: debug fuse = 0/0 2023-10-31T12:27:17.596 INFO:teuthology.orchestra.run.smithi061.stdout: debug heartbeatmap = 0/0 2023-10-31T12:27:17.596 INFO:teuthology.orchestra.run.smithi061.stdout: debug javaclient = 0/0 2023-10-31T12:27:17.596 INFO:teuthology.orchestra.run.smithi061.stdout: debug journal = 0/0 2023-10-31T12:27:17.596 INFO:teuthology.orchestra.run.smithi061.stdout: debug journaler = 0/0 2023-10-31T12:27:17.597 INFO:teuthology.orchestra.run.smithi061.stdout: debug kstore = 0/0 2023-10-31T12:27:17.597 INFO:teuthology.orchestra.run.smithi061.stdout: debug leveldb = 0/0 2023-10-31T12:27:17.597 INFO:teuthology.orchestra.run.smithi061.stdout: debug lockdep = 0/0 2023-10-31T12:27:17.597 INFO:teuthology.orchestra.run.smithi061.stdout: debug mds = 0/0 2023-10-31T12:27:17.597 INFO:teuthology.orchestra.run.smithi061.stdout: debug mds balancer = 0/0 2023-10-31T12:27:17.598 INFO:teuthology.orchestra.run.smithi061.stdout: debug mds locker = 0/0 2023-10-31T12:27:17.598 INFO:teuthology.orchestra.run.smithi061.stdout: debug mds log = 0/0 2023-10-31T12:27:17.598 INFO:teuthology.orchestra.run.smithi061.stdout: debug mds log expire = 0/0 2023-10-31T12:27:17.598 INFO:teuthology.orchestra.run.smithi061.stdout: debug mds migrator = 0/0 2023-10-31T12:27:17.598 INFO:teuthology.orchestra.run.smithi061.stdout: debug memdb = 0/0 2023-10-31T12:27:17.598 INFO:teuthology.orchestra.run.smithi061.stdout: debug mgr = 0/0 2023-10-31T12:27:17.599 INFO:teuthology.orchestra.run.smithi061.stdout: debug mgrc = 0/0 2023-10-31T12:27:17.599 INFO:teuthology.orchestra.run.smithi061.stdout: debug mon = 0/0 2023-10-31T12:27:17.599 INFO:teuthology.orchestra.run.smithi061.stdout: debug monc = 0/0 2023-10-31T12:27:17.599 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 0/0 2023-10-31T12:27:17.599 INFO:teuthology.orchestra.run.smithi061.stdout: debug objclass = 0/0 2023-10-31T12:27:17.600 INFO:teuthology.orchestra.run.smithi061.stdout: debug objectcacher = 0/0 2023-10-31T12:27:17.600 INFO:teuthology.orchestra.run.smithi061.stdout: debug objecter = 0/0 2023-10-31T12:27:17.600 INFO:teuthology.orchestra.run.smithi061.stdout: debug optracker = 0/0 2023-10-31T12:27:17.600 INFO:teuthology.orchestra.run.smithi061.stdout: debug osd = 0/0 2023-10-31T12:27:17.600 INFO:teuthology.orchestra.run.smithi061.stdout: debug paxos = 0/0 2023-10-31T12:27:17.601 INFO:teuthology.orchestra.run.smithi061.stdout: debug perfcounter = 0/0 2023-10-31T12:27:17.601 INFO:teuthology.orchestra.run.smithi061.stdout: debug rados = 0/0 2023-10-31T12:27:17.601 INFO:teuthology.orchestra.run.smithi061.stdout: debug rbd = 0/0 2023-10-31T12:27:17.601 INFO:teuthology.orchestra.run.smithi061.stdout: debug rbd mirror = 0/0 2023-10-31T12:27:17.601 INFO:teuthology.orchestra.run.smithi061.stdout: debug rbd replay = 0/0 2023-10-31T12:27:17.601 INFO:teuthology.orchestra.run.smithi061.stdout: debug refs = 0/0 2023-10-31T12:27:17.602 INFO:teuthology.orchestra.run.smithi061.stdout: debug rgw = 0/0 2023-10-31T12:27:17.602 INFO:teuthology.orchestra.run.smithi061.stdout: debug rgw sync = 0/0 2023-10-31T12:27:17.602 INFO:teuthology.orchestra.run.smithi061.stdout: debug rocksdb = 0/0 2023-10-31T12:27:17.602 INFO:teuthology.orchestra.run.smithi061.stdout: debug striper = 0/0 2023-10-31T12:27:17.602 INFO:teuthology.orchestra.run.smithi061.stdout: debug throttle = 0/0 2023-10-31T12:27:17.603 INFO:teuthology.orchestra.run.smithi061.stdout: debug timer = 0/0 2023-10-31T12:27:17.603 INFO:teuthology.orchestra.run.smithi061.stdout: debug tp = 0/0 2023-10-31T12:27:17.603 INFO:teuthology.orchestra.run.smithi061.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-10-31T12:27:17.603 INFO:teuthology.orchestra.run.smithi061.stdout: ms client mode = crc 2023-10-31T12:27:17.603 INFO:teuthology.orchestra.run.smithi061.stdout: ms cluster mode = crc 2023-10-31T12:27:17.604 INFO:teuthology.orchestra.run.smithi061.stdout: ms mon client mode = crc 2023-10-31T12:27:17.604 INFO:teuthology.orchestra.run.smithi061.stdout: ms mon cluster mode = crc 2023-10-31T12:27:17.604 INFO:teuthology.orchestra.run.smithi061.stdout: ms mon service mode = crc 2023-10-31T12:27:17.604 INFO:teuthology.orchestra.run.smithi061.stdout: ms service mode = crc 2023-10-31T12:27:17.604 INFO:teuthology.orchestra.run.smithi061.stdout: fsid = 30686b99-d4b7-4afe-9de4-18fff3a2aa8e 2023-10-31T12:27:17.604 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.605 INFO:teuthology.orchestra.run.smithi061.stdout:[osd] 2023-10-31T12:27:17.605 INFO:teuthology.orchestra.run.smithi061.stdout: osd journal size = 100 2023-10-31T12:27:17.605 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.605 INFO:teuthology.orchestra.run.smithi061.stdout: osd scrub load threshold = 5.0 2023-10-31T12:27:17.605 INFO:teuthology.orchestra.run.smithi061.stdout: osd scrub max interval = 600 2023-10-31T12:27:17.606 INFO:teuthology.orchestra.run.smithi061.stdout: osd mclock profile = high_recovery_ops 2023-10-31T12:27:17.606 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.606 INFO:teuthology.orchestra.run.smithi061.stdout: osd recover clone overlap = true 2023-10-31T12:27:17.606 INFO:teuthology.orchestra.run.smithi061.stdout: osd recovery max chunk = 1048576 2023-10-31T12:27:17.606 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.607 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug shutdown = true 2023-10-31T12:27:17.607 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug op order = true 2023-10-31T12:27:17.607 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug verify stray on activate = true 2023-10-31T12:27:17.607 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.607 INFO:teuthology.orchestra.run.smithi061.stdout: osd open classes on start = true 2023-10-31T12:27:17.608 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug pg log writeout = true 2023-10-31T12:27:17.608 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.608 INFO:teuthology.orchestra.run.smithi061.stdout: osd deep scrub update digest min age = 30 2023-10-31T12:27:17.608 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.608 INFO:teuthology.orchestra.run.smithi061.stdout: osd map max advance = 10 2023-10-31T12:27:17.609 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.609 INFO:teuthology.orchestra.run.smithi061.stdout: journal zero on create = true 2023-10-31T12:27:17.609 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.609 INFO:teuthology.orchestra.run.smithi061.stdout: filestore ondisk finisher threads = 3 2023-10-31T12:27:17.609 INFO:teuthology.orchestra.run.smithi061.stdout: filestore apply finisher threads = 3 2023-10-31T12:27:17.609 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.610 INFO:teuthology.orchestra.run.smithi061.stdout: bdev debug aio = true 2023-10-31T12:27:17.610 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug misdirected ops = true 2023-10-31T12:27:17.610 INFO:teuthology.orchestra.run.smithi061.stdout: debug monc = 20 2023-10-31T12:27:17.610 INFO:teuthology.orchestra.run.smithi061.stdout: bluestore block size = 96636764160 2023-10-31T12:27:17.610 INFO:teuthology.orchestra.run.smithi061.stdout: bluestore compression algorithm = snappy 2023-10-31T12:27:17.611 INFO:teuthology.orchestra.run.smithi061.stdout: bluestore compression mode = aggressive 2023-10-31T12:27:17.611 INFO:teuthology.orchestra.run.smithi061.stdout: bluestore fsck on mount = True 2023-10-31T12:27:17.611 INFO:teuthology.orchestra.run.smithi061.stdout: bluestore rocksdb cf = False 2023-10-31T12:27:17.612 INFO:teuthology.orchestra.run.smithi061.stdout: crimson seastar smp = 3 2023-10-31T12:27:17.612 INFO:teuthology.orchestra.run.smithi061.stdout: debug bluefs = 20 2023-10-31T12:27:17.612 INFO:teuthology.orchestra.run.smithi061.stdout: debug bluestore = 20 2023-10-31T12:27:17.613 INFO:teuthology.orchestra.run.smithi061.stdout: debug filestore = 0/0 2023-10-31T12:27:17.614 INFO:teuthology.orchestra.run.smithi061.stdout: debug journal = 0/0 2023-10-31T12:27:17.614 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 0/0 2023-10-31T12:27:17.614 INFO:teuthology.orchestra.run.smithi061.stdout: debug osd = 0/0 2023-10-31T12:27:17.615 INFO:teuthology.orchestra.run.smithi061.stdout: debug rocksdb = 10 2023-10-31T12:27:17.615 INFO:teuthology.orchestra.run.smithi061.stdout: err to stderr = True 2023-10-31T12:27:17.615 INFO:teuthology.orchestra.run.smithi061.stdout: log flush on exit = True 2023-10-31T12:27:17.615 INFO:teuthology.orchestra.run.smithi061.stdout: log to file = False 2023-10-31T12:27:17.615 INFO:teuthology.orchestra.run.smithi061.stdout: log to stderr = True 2023-10-31T12:27:17.616 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd backfillfull_ratio = 0.85 2023-10-31T12:27:17.616 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd full ratio = 0.9 2023-10-31T12:27:17.616 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd nearfull ratio = 0.8 2023-10-31T12:27:17.616 INFO:teuthology.orchestra.run.smithi061.stdout: osd failsafe full ratio = 0.95 2023-10-31T12:27:17.616 INFO:teuthology.orchestra.run.smithi061.stdout: osd objectstore = bluestore 2023-10-31T12:27:17.617 INFO:teuthology.orchestra.run.smithi061.stdout: osd shutdown pgref assert = True 2023-10-31T12:27:17.617 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.617 INFO:teuthology.orchestra.run.smithi061.stdout:[mgr] 2023-10-31T12:27:17.617 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2023-10-31T12:27:17.617 INFO:teuthology.orchestra.run.smithi061.stdout: debug mgr = 20 2023-10-31T12:27:17.618 INFO:teuthology.orchestra.run.smithi061.stdout: debug mon = 20 2023-10-31T12:27:17.618 INFO:teuthology.orchestra.run.smithi061.stdout: debug auth = 20 2023-10-31T12:27:17.618 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min pgs per osd = 4 2023-10-31T12:27:17.618 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min bytes per osd = 10 2023-10-31T12:27:17.618 INFO:teuthology.orchestra.run.smithi061.stdout: mgr/telemetry/nag = false 2023-10-31T12:27:17.619 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.619 INFO:teuthology.orchestra.run.smithi061.stdout:[mon] 2023-10-31T12:27:17.619 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 0/0 2023-10-31T12:27:17.619 INFO:teuthology.orchestra.run.smithi061.stdout: debug mon = 0/0 2023-10-31T12:27:17.619 INFO:teuthology.orchestra.run.smithi061.stdout: debug paxos = 0/0 2023-10-31T12:27:17.620 INFO:teuthology.orchestra.run.smithi061.stdout: debug auth = 20 2023-10-31T12:27:17.620 INFO:teuthology.orchestra.run.smithi061.stdout: mon data avail warn = 5 2023-10-31T12:27:17.620 INFO:teuthology.orchestra.run.smithi061.stdout: mon mgr mkfs grace = 240 2023-10-31T12:27:17.620 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min pgs per osd = 4 2023-10-31T12:27:17.620 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd reporter subtree level = osd 2023-10-31T12:27:17.621 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd prime pg temp = true 2023-10-31T12:27:17.621 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min bytes per osd = 10 2023-10-31T12:27:17.621 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.621 INFO:teuthology.orchestra.run.smithi061.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-31T12:27:17.621 INFO:teuthology.orchestra.run.smithi061.stdout: auth mon ticket ttl = 660 # 11m 2023-10-31T12:27:17.622 INFO:teuthology.orchestra.run.smithi061.stdout: auth service ticket ttl = 240 # 4m 2023-10-31T12:27:17.622 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.622 INFO:teuthology.orchestra.run.smithi061.stdout: # don't complain about insecure global_id in the test suite 2023-10-31T12:27:17.622 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-31T12:27:17.622 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-31T12:27:17.623 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.623 INFO:teuthology.orchestra.run.smithi061.stdout: # 1m isn't quite enough 2023-10-31T12:27:17.623 INFO:teuthology.orchestra.run.smithi061.stdout: mon_down_mkfs_grace = 2m 2023-10-31T12:27:17.623 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.623 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_filestore_osds = false 2023-10-31T12:27:17.623 INFO:teuthology.orchestra.run.smithi061.stdout: osd pool default crimson = True 2023-10-31T12:27:17.624 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:17.624 INFO:teuthology.orchestra.run.smithi061.stdout:[client] 2023-10-31T12:27:17.624 INFO:teuthology.orchestra.run.smithi061.stdout: rgw cache enabled = true 2023-10-31T12:27:17.624 INFO:teuthology.orchestra.run.smithi061.stdout: rgw enable ops log = true 2023-10-31T12:27:17.624 INFO:teuthology.orchestra.run.smithi061.stdout: rgw enable usage log = true 2023-10-31T12:27:17.625 INFO:teuthology.orchestra.run.smithi061.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-31T12:27:17.625 INFO:teuthology.orchestra.run.smithi061.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-31T12:27:17.625 INFO:teuthology.orchestra.run.smithi061.stdout:[mon.a] 2023-10-31T12:27:17.626 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-31T12:27:17.627 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-31T12:27:17.692 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-31T12:27:17.692 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:27:17.693 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-31T12:27:17.707 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:27:17.708 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-31T12:27:17.763 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:27:17.763 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:27:17.764 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-31T12:27:17.764 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-31T12:27:17.850 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:27:17.850 DEBUG:teuthology.orchestra.run.smithi061:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-31T12:27:17.905 INFO:tasks.ceph:Setting up mon nodes... 2023-10-31T12:27:17.906 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-31T12:27:17.906 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-31T12:27:18.012 INFO:teuthology.orchestra.run.smithi061.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-31T12:27:18.015 INFO:tasks.ceph:Setting up mds nodes... 2023-10-31T12:27:18.015 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-31T12:27:18.016 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-31T12:27:18.067 INFO:teuthology.orchestra.run.smithi061.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-31T12:27:18.086 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-31T12:27:18.086 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi061.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-31T12:27:18.087 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-31T12:27:18.117 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-31T12:27:18.117 INFO:tasks.ceph:role: osd.0 2023-10-31T12:27:18.117 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:27:18.118 DEBUG:teuthology.orchestra.run.smithi061:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-31T12:27:18.195 INFO:teuthology.orchestra.run.smithi061.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-31T12:27:18.195 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-31T12:27:18.195 INFO:teuthology.orchestra.run.smithi061.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-31T12:27:18.196 INFO:teuthology.orchestra.run.smithi061.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-31T12:27:18.196 INFO:teuthology.orchestra.run.smithi061.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-31T12:27:18.196 INFO:teuthology.orchestra.run.smithi061.stdout: = sunit=0 swidth=0 blks 2023-10-31T12:27:18.196 INFO:teuthology.orchestra.run.smithi061.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-31T12:27:18.196 INFO:teuthology.orchestra.run.smithi061.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-31T12:27:18.197 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-31T12:27:18.197 INFO:teuthology.orchestra.run.smithi061.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-31T12:27:18.718 INFO:teuthology.orchestra.run.smithi061.stdout:Discarding blocks...Done. 2023-10-31T12:27:18.720 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi061.front.sepia.ceph.com -o noatime 2023-10-31T12:27:18.720 DEBUG:teuthology.orchestra.run.smithi061:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-31T12:27:18.847 DEBUG:teuthology.orchestra.run.smithi061:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-31T12:27:18.888 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-31T12:27:18.960 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-31T12:27:18.961 INFO:tasks.ceph:role: osd.1 2023-10-31T12:27:18.961 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:27:18.962 DEBUG:teuthology.orchestra.run.smithi061:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-31T12:27:19.040 INFO:teuthology.orchestra.run.smithi061.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-31T12:27:19.040 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-31T12:27:19.041 INFO:teuthology.orchestra.run.smithi061.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-31T12:27:19.041 INFO:teuthology.orchestra.run.smithi061.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-31T12:27:19.041 INFO:teuthology.orchestra.run.smithi061.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-31T12:27:19.041 INFO:teuthology.orchestra.run.smithi061.stdout: = sunit=0 swidth=0 blks 2023-10-31T12:27:19.042 INFO:teuthology.orchestra.run.smithi061.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-31T12:27:19.042 INFO:teuthology.orchestra.run.smithi061.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-31T12:27:19.042 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-31T12:27:19.042 INFO:teuthology.orchestra.run.smithi061.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-31T12:27:19.485 INFO:teuthology.orchestra.run.smithi061.stdout:Discarding blocks...Done. 2023-10-31T12:27:19.487 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi061.front.sepia.ceph.com -o noatime 2023-10-31T12:27:19.487 DEBUG:teuthology.orchestra.run.smithi061:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-31T12:27:19.581 DEBUG:teuthology.orchestra.run.smithi061:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-31T12:27:19.618 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-31T12:27:19.689 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-31T12:27:19.690 INFO:tasks.ceph:role: osd.2 2023-10-31T12:27:19.690 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:27:19.691 DEBUG:teuthology.orchestra.run.smithi061:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-31T12:27:19.768 INFO:teuthology.orchestra.run.smithi061.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-31T12:27:19.769 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-31T12:27:19.769 INFO:teuthology.orchestra.run.smithi061.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-31T12:27:19.769 INFO:teuthology.orchestra.run.smithi061.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-31T12:27:19.769 INFO:teuthology.orchestra.run.smithi061.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-31T12:27:19.770 INFO:teuthology.orchestra.run.smithi061.stdout: = sunit=0 swidth=0 blks 2023-10-31T12:27:19.770 INFO:teuthology.orchestra.run.smithi061.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-31T12:27:19.770 INFO:teuthology.orchestra.run.smithi061.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-31T12:27:19.770 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-31T12:27:19.770 INFO:teuthology.orchestra.run.smithi061.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-31T12:27:20.260 INFO:teuthology.orchestra.run.smithi061.stdout:Discarding blocks...Done. 2023-10-31T12:27:20.262 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi061.front.sepia.ceph.com -o noatime 2023-10-31T12:27:20.263 DEBUG:teuthology.orchestra.run.smithi061:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-31T12:27:20.391 DEBUG:teuthology.orchestra.run.smithi061:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-31T12:27:20.431 DEBUG:teuthology.orchestra.run.smithi061:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-31T12:27:20.982 INFO:teuthology.orchestra.run.smithi061.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-31T12:27:20.983 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:20,982 seastar - Reactor backend: linux-aio 2023-10-31T12:27:20.999 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:20,998 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-31T12:27:20.999 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:20,998 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-31T12:27:20.999 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:20,998 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-31T12:27:21.000 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:20,998 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-31T12:27:21.001 INFO:teuthology.orchestra.run.smithi061.stderr:==80226==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-31T12:27:21.007 INFO:teuthology.orchestra.run.smithi061.stderr:==80226==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe3953e000; bottom 0x7fdb52992000; size: 0x0022e6bac000 (149899886592) 2023-10-31T12:27:21.008 INFO:teuthology.orchestra.run.smithi061.stderr:False positive error reports may follow 2023-10-31T12:27:21.008 INFO:teuthology.orchestra.run.smithi061.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-31T12:27:21.033 INFO:teuthology.orchestra.run.smithi061.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c63c74a 0x2c63daf8 0x2c63f046 0x2c63f3ca 0x2c63f514 0x2c63f96a 0x54daf 0x22c529d0 0x22c62199 0x22c63895 0x22c6426e 0x22c65359 0x22d37d18 0x22baeb39 0x22c761e4 0x22c77c2f 0x22c78998 0x22c40804 0x22c3fe5c 0x22971c62 0x24973b31 0x24979259 0x2497b7f4 0x2497c9cd 0x224fd1f1 0x22538a7e 0x22560df5 0x2c5dea65 0x2c67c872 0x2c8a0672 0x2c8a24bd 0x2c2faad6 0x2c2fc9d7 0x1ed6c220 0x1ed7638d 0x1ebf325d 0x3feaf 0x3ff5f 0x1e6ada04 2023-10-31T12:27:21.034 INFO:teuthology.orchestra.run.smithi061.stderr:kernel callstack: 2023-10-31T12:27:21.550 INFO:teuthology.orchestra.run.smithi061.stderr:----------------------------------------------------- 2023-10-31T12:27:21.551 INFO:teuthology.orchestra.run.smithi061.stderr:Suppressions used: 2023-10-31T12:27:21.551 INFO:teuthology.orchestra.run.smithi061.stderr: count bytes template 2023-10-31T12:27:21.551 INFO:teuthology.orchestra.run.smithi061.stderr: 1 8 MallocExtension::Register 2023-10-31T12:27:21.551 INFO:teuthology.orchestra.run.smithi061.stderr:----------------------------------------------------- 2023-10-31T12:27:21.551 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-10-31T12:27:21.601 INFO:teuthology.orchestra.run.smithi061.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-31T12:27:21.601 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:21,600 seastar - Reactor backend: linux-aio 2023-10-31T12:27:21.629 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:21,628 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-31T12:27:21.629 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:21,628 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-31T12:27:21.629 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:21,628 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-31T12:27:21.637 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:21,635 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-31T12:27:21.639 INFO:teuthology.orchestra.run.smithi061.stderr:==80224==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-31T12:27:21.645 INFO:teuthology.orchestra.run.smithi061.stderr:==80224==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe3953e000; bottom 0x7fdb5077b000; size: 0x0022e8dc3000 (149935632384) 2023-10-31T12:27:21.645 INFO:teuthology.orchestra.run.smithi061.stderr:False positive error reports may follow 2023-10-31T12:27:21.645 INFO:teuthology.orchestra.run.smithi061.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-31T12:27:21.671 INFO:teuthology.orchestra.run.smithi061.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c63c74a 0x2c63daf8 0x2c63f046 0x2c63f3ca 0x2c63f514 0x2c63f96a 0x54daf 0x20b7947f 0x2497c1ce 0x2497c9cd 0x224fd1f1 0x22538a7e 0x22560df5 0x2c5dea65 0x2c67c872 0x2c8a0672 0x2c8a24bd 0x2c2faad6 0x2c2fc9d7 0x1ebf4b13 0x3feaf 0x3ff5f 0x1e6ada04 2023-10-31T12:27:21.671 INFO:teuthology.orchestra.run.smithi061.stderr:kernel callstack: 2023-10-31T12:27:21.731 INFO:teuthology.orchestra.run.smithi061.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c63c74a 0x2c63daf8 0x2c63f046 0x2c63f3ca 0x2c63f514 0x2c63f96a 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x261e362a 0x261e5c73 0x261e7a84 0x261ea2e9 0x2601a24d 0x1ecae2c0 0x1ec23237 0x1ec271a3 0x2c9c039d 2023-10-31T12:27:21.732 INFO:teuthology.orchestra.run.smithi061.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa2a62817 0xffffffffa2a5e167 0xffffffffa2c000aa 2023-10-31T12:27:25.695 INFO:teuthology.orchestra.run.smithi061.stderr:----------------------------------------------------- 2023-10-31T12:27:25.695 INFO:teuthology.orchestra.run.smithi061.stderr:Suppressions used: 2023-10-31T12:27:25.696 INFO:teuthology.orchestra.run.smithi061.stderr: count bytes template 2023-10-31T12:27:25.696 INFO:teuthology.orchestra.run.smithi061.stderr: 1 8 MallocExtension::Register 2023-10-31T12:27:25.696 INFO:teuthology.orchestra.run.smithi061.stderr:----------------------------------------------------- 2023-10-31T12:27:25.696 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-10-31T12:27:25.697 INFO:teuthology.orchestra.run.smithi061.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 30686b99-d4b7-4afe-9de4-18fff3a2aa8e 2023-10-31T12:27:25.735 DEBUG:teuthology.orchestra.run.smithi061:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-31T12:27:25.766 DEBUG:teuthology.orchestra.run.smithi061:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-31T12:27:26.311 INFO:teuthology.orchestra.run.smithi061.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-31T12:27:26.312 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:26,311 seastar - Reactor backend: linux-aio 2023-10-31T12:27:26.329 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:26,327 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-31T12:27:26.329 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:26,327 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-31T12:27:26.329 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:26,327 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-31T12:27:26.329 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:26,328 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-31T12:27:26.331 INFO:teuthology.orchestra.run.smithi061.stderr:==80408==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-31T12:27:26.337 INFO:teuthology.orchestra.run.smithi061.stderr:==80408==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffedf4c1000; bottom 0x7fd315d55000; size: 0x002bc976c000 (188063596544) 2023-10-31T12:27:26.338 INFO:teuthology.orchestra.run.smithi061.stderr:False positive error reports may follow 2023-10-31T12:27:26.338 INFO:teuthology.orchestra.run.smithi061.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-31T12:27:26.363 INFO:teuthology.orchestra.run.smithi061.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c63c74a 0x2c63daf8 0x2c63f046 0x2c63f3ca 0x2c63f514 0x2c63f96a 0x54daf 0x22c9d2fe 0x22c9fd43 0x22ca0075 0x22ca172f 0x22ca4596 0x22ca4a89 0x22c52b79 0x22c62199 0x22c63895 0x22c6426e 0x22c65359 0x22d37d18 0x22baeb39 0x22c761e4 0x22c77c2f 0x22c78998 0x22c40804 0x22c3fe5c 0x22971c62 0x24973b31 0x24979259 0x2497b7f4 0x2497c9cd 0x224fd1f1 0x22538a7e 0x22560df5 0x2c5dea65 0x2c67c872 0x2c8a0672 0x2c8a24bd 0x2c2faad6 0x2c2fc9d7 0x1ed6c220 0x1ed7638d 0x1ebf325d 0x3feaf 0x3ff5f 0x1e6ada04 2023-10-31T12:27:26.363 INFO:teuthology.orchestra.run.smithi061.stderr:kernel callstack: 2023-10-31T12:27:26.876 INFO:teuthology.orchestra.run.smithi061.stderr:----------------------------------------------------- 2023-10-31T12:27:26.876 INFO:teuthology.orchestra.run.smithi061.stderr:Suppressions used: 2023-10-31T12:27:26.877 INFO:teuthology.orchestra.run.smithi061.stderr: count bytes template 2023-10-31T12:27:26.877 INFO:teuthology.orchestra.run.smithi061.stderr: 1 8 MallocExtension::Register 2023-10-31T12:27:26.877 INFO:teuthology.orchestra.run.smithi061.stderr:----------------------------------------------------- 2023-10-31T12:27:26.877 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-10-31T12:27:26.932 INFO:teuthology.orchestra.run.smithi061.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-31T12:27:26.932 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:26,932 seastar - Reactor backend: linux-aio 2023-10-31T12:27:26.968 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:26,967 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-31T12:27:26.968 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:26,967 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-31T12:27:26.968 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:26,967 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-31T12:27:26.969 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:26,967 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-31T12:27:26.978 INFO:teuthology.orchestra.run.smithi061.stderr:==80406==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-31T12:27:26.984 INFO:teuthology.orchestra.run.smithi061.stderr:==80406==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffedf4c1000; bottom 0x7fd313b3e000; size: 0x002bcb983000 (188099342336) 2023-10-31T12:27:26.984 INFO:teuthology.orchestra.run.smithi061.stderr:False positive error reports may follow 2023-10-31T12:27:26.984 INFO:teuthology.orchestra.run.smithi061.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-31T12:27:27.010 INFO:teuthology.orchestra.run.smithi061.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c63c74a 0x2c63daf8 0x2c63f046 0x2c63f3ca 0x2c63f514 0x2c63f96a 0x54daf 0x228f1241 0x228f6dcf 0x228f878e 0x249745fe 0x24979259 0x2497b7f4 0x2497c9cd 0x224fd1f1 0x22538a7e 0x22560df5 0x2c5dea65 0x2c67c872 0x2c8a0672 0x2c8a24bd 0x2c2faad6 0x2c2fc9d7 0x1ebf4b13 0x3feaf 0x3ff5f 0x1e6ada04 2023-10-31T12:27:27.010 INFO:teuthology.orchestra.run.smithi061.stderr:kernel callstack: 2023-10-31T12:27:31.286 INFO:teuthology.orchestra.run.smithi061.stderr:----------------------------------------------------- 2023-10-31T12:27:31.287 INFO:teuthology.orchestra.run.smithi061.stderr:Suppressions used: 2023-10-31T12:27:31.287 INFO:teuthology.orchestra.run.smithi061.stderr: count bytes template 2023-10-31T12:27:31.287 INFO:teuthology.orchestra.run.smithi061.stderr: 1 8 MallocExtension::Register 2023-10-31T12:27:31.287 INFO:teuthology.orchestra.run.smithi061.stderr:----------------------------------------------------- 2023-10-31T12:27:31.288 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-10-31T12:27:31.288 INFO:teuthology.orchestra.run.smithi061.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid 30686b99-d4b7-4afe-9de4-18fff3a2aa8e 2023-10-31T12:27:31.342 DEBUG:teuthology.orchestra.run.smithi061:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-31T12:27:31.373 DEBUG:teuthology.orchestra.run.smithi061:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-31T12:27:31.918 INFO:teuthology.orchestra.run.smithi061.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-31T12:27:31.919 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:31,918 seastar - Reactor backend: linux-aio 2023-10-31T12:27:31.936 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:31,934 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-31T12:27:31.936 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:31,934 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-31T12:27:31.936 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:31,934 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-31T12:27:31.936 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:31,935 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-31T12:27:31.938 INFO:teuthology.orchestra.run.smithi061.stderr:==80590==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-31T12:27:31.944 INFO:teuthology.orchestra.run.smithi061.stderr:==80590==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc72e51000; bottom 0x7f040dd3e000; size: 0x00f865113000 (1066847514624) 2023-10-31T12:27:31.944 INFO:teuthology.orchestra.run.smithi061.stderr:False positive error reports may follow 2023-10-31T12:27:31.945 INFO:teuthology.orchestra.run.smithi061.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-31T12:27:31.970 INFO:teuthology.orchestra.run.smithi061.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c63c74a 0x2c63daf8 0x2c63f046 0x2c63f3ca 0x2c63f514 0x2c63f96a 0x54daf 0x22bbf456 0x22c48d7f 0x22c4da08 0x22c4df56 0x22c42d62 0x22c44c2b 0x22c5d7d7 0x22c60768 0x22c84f8c 0x22ca4201 0x22ca4a89 0x22c52b79 0x22c62199 0x22c63895 0x22c6426e 0x22c65359 0x22d37d18 0x22baeb39 0x22c761e4 0x22c77c2f 0x22c78998 0x22c40804 0x22c3fe5c 0x22971c62 0x24973b31 0x24979259 0x2497b7f4 0x2497c9cd 0x224fd1f1 0x22538a7e 0x22560df5 0x2c5dea65 0x2c67c872 0x2c8a0672 0x2c8a24bd 0x2c2faad6 0x2c2fc9d7 0x1ed6c220 0x1ed7638d 0x1ebf325d 0x3feaf 0x3ff5f 0x1e6ada04 2023-10-31T12:27:31.970 INFO:teuthology.orchestra.run.smithi061.stderr:kernel callstack: 2023-10-31T12:27:32.487 INFO:teuthology.orchestra.run.smithi061.stderr:----------------------------------------------------- 2023-10-31T12:27:32.487 INFO:teuthology.orchestra.run.smithi061.stderr:Suppressions used: 2023-10-31T12:27:32.487 INFO:teuthology.orchestra.run.smithi061.stderr: count bytes template 2023-10-31T12:27:32.488 INFO:teuthology.orchestra.run.smithi061.stderr: 1 8 MallocExtension::Register 2023-10-31T12:27:32.488 INFO:teuthology.orchestra.run.smithi061.stderr:----------------------------------------------------- 2023-10-31T12:27:32.488 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-10-31T12:27:32.543 INFO:teuthology.orchestra.run.smithi061.stderr:WARNING: debug mode. Not for benchmarking or production 2023-10-31T12:27:32.543 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:32,542 seastar - Reactor backend: linux-aio 2023-10-31T12:27:32.561 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:32,560 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-10-31T12:27:32.562 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:32,560 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-10-31T12:27:32.562 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:32,560 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-10-31T12:27:32.569 INFO:teuthology.orchestra.run.smithi061.stderr:INFO 2023-10-31 12:27:32,568 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-10-31T12:27:32.571 INFO:teuthology.orchestra.run.smithi061.stderr:==80588==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-10-31T12:27:32.577 INFO:teuthology.orchestra.run.smithi061.stderr:==80588==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc72e51000; bottom 0x7f040bb27000; size: 0x00f86732a000 (1066883260416) 2023-10-31T12:27:32.578 INFO:teuthology.orchestra.run.smithi061.stderr:False positive error reports may follow 2023-10-31T12:27:32.578 INFO:teuthology.orchestra.run.smithi061.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-10-31T12:27:32.661 INFO:teuthology.orchestra.run.smithi061.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c63c74a 0x2c63daf8 0x2c63f046 0x2c63f3ca 0x2c63f514 0x2c63f96a 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x261e362a 0x261e5c73 0x261e7a84 0x261ea2e9 0x2601a24d 0x1ecae2c0 0x1ec23237 0x1ec271a3 0x2c9c039d 2023-10-31T12:27:32.662 INFO:teuthology.orchestra.run.smithi061.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa1f4ffa3 0xffffffffa1f4ffea 0xffffffffa2a5e18c 0xffffffffa2c000aa 2023-10-31T12:27:36.613 INFO:teuthology.orchestra.run.smithi061.stderr:----------------------------------------------------- 2023-10-31T12:27:36.613 INFO:teuthology.orchestra.run.smithi061.stderr:Suppressions used: 2023-10-31T12:27:36.613 INFO:teuthology.orchestra.run.smithi061.stderr: count bytes template 2023-10-31T12:27:36.613 INFO:teuthology.orchestra.run.smithi061.stderr: 1 8 MallocExtension::Register 2023-10-31T12:27:36.613 INFO:teuthology.orchestra.run.smithi061.stderr:----------------------------------------------------- 2023-10-31T12:27:36.614 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-10-31T12:27:36.614 INFO:teuthology.orchestra.run.smithi061.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid 30686b99-d4b7-4afe-9de4-18fff3a2aa8e 2023-10-31T12:27:36.651 DEBUG:teuthology.orchestra.run.smithi061:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-31T12:27:36.681 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-31T12:27:36.682 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:27:36.682 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-31T12:27:36.751 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:27:36.752 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-31T12:27:36.821 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:27:36.822 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-31T12:27:36.891 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:27:36.892 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-31T12:27:36.961 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:27:36.961 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-31T12:27:37.016 INFO:tasks.ceph:Adding keys to all mons... 2023-10-31T12:27:37.017 DEBUG:teuthology.orchestra.run.smithi061:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-31T12:27:37.085 INFO:teuthology.orchestra.run.smithi061.stdout:[mgr.x] 2023-10-31T12:27:37.085 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQCm8kBl5zuxABAApyGdP9/24hmNBrFvG4UrGA== 2023-10-31T12:27:37.085 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.0] 2023-10-31T12:27:37.086 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQCp8kBlntpiKRAA7yRC3Yj5+2S0kR1gBFvkbQ== 2023-10-31T12:27:37.086 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.1] 2023-10-31T12:27:37.086 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQCv8kBlDkvzARAAtokRuokFyhaX+FiBy5QaZA== 2023-10-31T12:27:37.086 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.2] 2023-10-31T12:27:37.087 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQC08kBlMKZMJRAAbS0SYWGdc/DzAtDNjWdUXw== 2023-10-31T12:27:37.087 INFO:teuthology.orchestra.run.smithi061.stdout:[client.0] 2023-10-31T12:27:37.087 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQCm8kBlBZn4AxAALQVx2E6YZi9daLKSpcthlw== 2023-10-31T12:27:37.088 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-31T12:27:37.190 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-31T12:27:37.253 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-31T12:27:37.315 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-31T12:27:37.377 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-31T12:27:37.442 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-31T12:27:37.443 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-31T12:27:37.473 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-31T12:27:37.577 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:37.575+0000 7f87d87d9980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:37.578 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:37.575+0000 7f87d87d9980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:37.593 DEBUG:teuthology.orchestra.run.smithi061:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-31T12:27:37.623 DEBUG:teuthology.orchestra.run.smithi061:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-31T12:27:37.678 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-31T12:27:37.679 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-31T12:27:37.679 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-31T12:27:37.720 INFO:tasks.ceph.mon.a:Started 2023-10-31T12:27:37.721 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-31T12:27:37.721 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-31T12:27:37.721 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-31T12:27:37.723 INFO:tasks.ceph.mgr.x:Started 2023-10-31T12:27:37.724 INFO:tasks.ceph:Setting crush tunables to default 2023-10-31T12:27:37.724 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph --cluster ceph osd crush tunables default 2023-10-31T12:27:37.807 INFO:tasks.ceph.mon.a.smithi061.stderr:2023-10-31T12:27:37.805+0000 7f9db5c3f980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:37.808 INFO:tasks.ceph.mon.a.smithi061.stderr:2023-10-31T12:27:37.805+0000 7f9db5c3f980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:37.808 INFO:tasks.ceph.mon.a.smithi061.stderr:2023-10-31T12:27:37.805+0000 7f9db5c3f980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:37.809 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:37.807+0000 7fcca616df00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:37.837 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:37.830+0000 7f37a146d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:37.861 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:37.859+0000 7f37a146d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:37.931 INFO:teuthology.orchestra.run.smithi061.stderr:adjusted tunables profile to default 2023-10-31T12:27:37.942 INFO:tasks.ceph:check_enable_crimson: True 2023-10-31T12:27:37.942 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2023-10-31T12:27:37.943 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2023-10-31T12:27:38.013 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:38.010+0000 7fcca616df00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:38.013 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:38.011+0000 7fcca616df00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:38.052 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:38.046+0000 7f41de464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:38.054 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:38.052+0000 7f41de464640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:38.147 INFO:tasks.ceph.mon.a.smithi061.stderr:2023-10-31T12:27:38.145+0000 7f9dacff1640 -1 WARNING: experimental feature 'crimson' is enabled 2023-10-31T12:27:38.148 INFO:tasks.ceph.mon.a.smithi061.stderr:Please be aware that this feature is experimental, untested, 2023-10-31T12:27:38.148 INFO:tasks.ceph.mon.a.smithi061.stderr:unsupported, and may result in data corruption, data loss, 2023-10-31T12:27:38.148 INFO:tasks.ceph.mon.a.smithi061.stderr:and/or irreparable damage to your cluster. Do not use 2023-10-31T12:27:38.148 INFO:tasks.ceph.mon.a.smithi061.stderr:feature with important data. 2023-10-31T12:27:38.149 INFO:tasks.ceph.mon.a.smithi061.stderr: 2023-10-31T12:27:38.484 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:38.482+0000 7fcca616df00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-31T12:27:38.538 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:38.535+0000 7fcca616df00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-31T12:27:38.943 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-31T12:27:38.944 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:27:38.944 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-31T12:27:38.975 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:27:38.976 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-31T12:27:39.046 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:27:39.047 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-31T12:27:39.096 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:39.094+0000 7fcca616df00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-31T12:27:39.117 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph --cluster ceph osd new cfb39abb-ee2e-48ae-b526-9bf48052c2f4 0 2023-10-31T12:27:39.152 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:39.149+0000 7fcca616df00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-31T12:27:39.200 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:39.197+0000 7fcca616df00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-31T12:27:39.258 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:39.256+0000 7fcca616df00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-31T12:27:39.262 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:39.255+0000 7f863491e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:39.264 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:39.262+0000 7f863491e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:39.303 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:39.301+0000 7fcca616df00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-31T12:27:39.342 INFO:teuthology.orchestra.run.smithi061.stdout:0 2023-10-31T12:27:39.353 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph --cluster ceph osd new 60ad73ed-e8f8-45eb-9628-28e316f03d77 1 2023-10-31T12:27:39.411 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:39.409+0000 7fcca616df00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-31T12:27:39.463 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:39.455+0000 7f97f359e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:39.465 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:39.462+0000 7f97f359e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:39.487 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:39.485+0000 7fcca616df00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-31T12:27:39.535 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:39.533+0000 7fcca616df00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-31T12:27:39.543 INFO:teuthology.orchestra.run.smithi061.stdout:1 2023-10-31T12:27:39.554 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph --cluster ceph osd new 756e2aa5-706f-43b3-9645-d8fd314d24ad 2 2023-10-31T12:27:39.593 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:39.590+0000 7fcca616df00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-31T12:27:39.644 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:39.641+0000 7fcca616df00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-31T12:27:39.664 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:39.657+0000 7ff35b19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:39.666 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:39.664+0000 7ff35b19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:39.743 INFO:teuthology.orchestra.run.smithi061.stdout:2 2023-10-31T12:27:39.754 DEBUG:teuthology.orchestra.run.smithi061:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2023-10-31T12:27:39.830 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-31T12:27:39.830 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2023-10-31T12:27:39.873 INFO:tasks.ceph.osd.0:Started 2023-10-31T12:27:39.873 DEBUG:teuthology.orchestra.run.smithi061:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2023-10-31T12:27:39.909 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-31T12:27:39.910 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2023-10-31T12:27:39.951 INFO:tasks.ceph.osd.1:Started 2023-10-31T12:27:39.952 DEBUG:teuthology.orchestra.run.smithi061:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2023-10-31T12:27:39.990 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-31T12:27:39.991 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2023-10-31T12:27:40.033 INFO:tasks.ceph.osd.2:Started 2023-10-31T12:27:40.033 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-31T12:27:40.075 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:40.073+0000 7fcca616df00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-31T12:27:40.166 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:40.159+0000 7fc3edcde640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:40.168 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:40.166+0000 7fc3edcde640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:40.240 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:40.241 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":6,"fsid":"30686b99-d4b7-4afe-9de4-18fff3a2aa8e","created":"2023-10-31T12:27:37.848465+0000","modified":"2023-10-31T12:27:39.741843+0000","last_up_change":"0.000000","last_in_change":"2023-10-31T12:27:39.741843+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"cfb39abb-ee2e-48ae-b526-9bf48052c2f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"60ad73ed-e8f8-45eb-9628-28e316f03d77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"756e2aa5-706f-43b3-9645-d8fd314d24ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-31T12:27:40.243 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:40.241+0000 7fcca616df00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-31T12:27:40.267 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-31T12:27:40.267 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-31T12:27:40.390 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:40.387+0000 7fcca616df00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-31T12:27:40.447 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:40.445+0000 7fcca616df00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-31T12:27:40.558 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:40.555+0000 7fcca616df00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-31T12:27:40.569 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-31T12:27:40.666 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:40.663+0000 7fcca616df00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-31T12:27:40.670 INFO:teuthology.misc.health.smithi061.stderr:2023-10-31T12:27:40.665+0000 7f65e979e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:40.672 INFO:teuthology.misc.health.smithi061.stderr:2023-10-31T12:27:40.670+0000 7f65e979e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:40.740 INFO:teuthology.misc.health.smithi061.stdout: 2023-10-31T12:27:40.741 INFO:teuthology.misc.health.smithi061.stdout:{"epoch":6,"fsid":"30686b99-d4b7-4afe-9de4-18fff3a2aa8e","created":"2023-10-31T12:27:37.848465+0000","modified":"2023-10-31T12:27:39.741843+0000","last_up_change":"0.000000","last_in_change":"2023-10-31T12:27:39.741843+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"cfb39abb-ee2e-48ae-b526-9bf48052c2f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"60ad73ed-e8f8-45eb-9628-28e316f03d77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"756e2aa5-706f-43b3-9645-d8fd314d24ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-31T12:27:40.750 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-31T12:27:40.820 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:40.817+0000 7fcca616df00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-31T12:27:40.867 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:40.865+0000 7fcca616df00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-31T12:27:41.022 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:41.019+0000 7fcca616df00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-31T12:27:41.883 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:41.881+0000 7fcca616df00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-31T12:27:42.177 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:42.175+0000 7fcca616df00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-31T12:27:42.232 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:42.230+0000 7fcca616df00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-31T12:27:42.339 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:42.337+0000 7fcca616df00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-31T12:27:43.360 INFO:tasks.ceph.mgr.x.smithi061.stderr:2023-10-31T12:27:43.357+0000 7fcc45d27640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-31T12:27:47.052 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-31T12:27:47.163 INFO:teuthology.misc.health.smithi061.stderr:2023-10-31T12:27:47.158+0000 7f04d2aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:47.165 INFO:teuthology.misc.health.smithi061.stderr:2023-10-31T12:27:47.163+0000 7f04d2aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:47.349 INFO:teuthology.misc.health.smithi061.stdout: 2023-10-31T12:27:47.350 INFO:teuthology.misc.health.smithi061.stdout:{"epoch":8,"fsid":"30686b99-d4b7-4afe-9de4-18fff3a2aa8e","created":"2023-10-31T12:27:37.848465+0000","modified":"2023-10-31T12:27:45.358885+0000","last_up_change":"2023-10-31T12:27:45.358885+0000","last_in_change":"2023-10-31T12:27:39.741843+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"cfb39abb-ee2e-48ae-b526-9bf48052c2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6803","nonce":2536790504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":2536790504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":2536790504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6809","nonce":2536790504}]},"public_addr":"172.21.15.61:6803/2536790504","cluster_addr":"172.21.15.61:6802/2536790504","heartbeat_back_addr":"172.21.15.61:6808/2536790504","heartbeat_front_addr":"172.21.15.61:6809/2536790504","state":["exists","up"]},{"osd":1,"uuid":"60ad73ed-e8f8-45eb-9628-28e316f03d77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":2624151341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6805","nonce":2624151341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":2624151341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6811","nonce":2624151341}]},"public_addr":"172.21.15.61:6804/2624151341","cluster_addr":"172.21.15.61:6805/2624151341","heartbeat_back_addr":"172.21.15.61:6810/2624151341","heartbeat_front_addr":"172.21.15.61:6811/2624151341","state":["exists","up"]},{"osd":2,"uuid":"756e2aa5-706f-43b3-9645-d8fd314d24ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":3069982037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6807","nonce":3069982037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":3069982037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6813","nonce":3069982037}]},"public_addr":"172.21.15.61:6806/3069982037","cluster_addr":"172.21.15.61:6807/3069982037","heartbeat_back_addr":"172.21.15.61:6812/3069982037","heartbeat_front_addr":"172.21.15.61:6813/3069982037","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-31T12:27:47.360 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-31T12:27:47.360 INFO:tasks.ceph:Creating RBD pool 2023-10-31T12:27:47.361 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-31T12:27:47.476 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:47.468+0000 7f9857f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:47.478 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:47.475+0000 7f9857f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:48.374 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'rbd' created 2023-10-31T12:27:48.392 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-31T12:27:48.520 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:48.509+0000 7f369f02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:48.522 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:48.519+0000 7f369f02d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:49.383 INFO:teuthology.orchestra.run.smithi061.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-31T12:27:49.398 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-31T12:27:49.398 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-31T12:27:49.398 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-31T12:27:49.399 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-31T12:27:49.399 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-31T12:27:49.530 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:49.524+0000 7f76c0155640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:49.532 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:49.530+0000 7f76c0155640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:49.769 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:49.780 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":2758386111},{"type":"v1","addr":"172.21.15.61:6801","nonce":2758386111}]},"active_addr":"172.21.15.61:6801/2758386111","active_change":"2023-10-31T12:27:42.351367+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.61:0","nonce":1688755272}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.61:0","nonce":865844384}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.61:0","nonce":3003976504}]}]} 2023-10-31T12:27:49.782 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-31T12:27:49.783 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-31T12:27:49.783 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-31T12:27:49.894 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:49.889+0000 7f41f506d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:49.896 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:49.894+0000 7f41f506d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:50.083 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:50.084 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":11,"fsid":"30686b99-d4b7-4afe-9de4-18fff3a2aa8e","created":"2023-10-31T12:27:37.848465+0000","modified":"2023-10-31T12:27:49.378814+0000","last_up_change":"2023-10-31T12:27:45.358885+0000","last_in_change":"2023-10-31T12:27:39.741843+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-31T12:27:46.410809+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-31T12:27:47.689691+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfb39abb-ee2e-48ae-b526-9bf48052c2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6803","nonce":2536790504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":2536790504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":2536790504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6809","nonce":2536790504}]},"public_addr":"172.21.15.61:6803/2536790504","cluster_addr":"172.21.15.61:6802/2536790504","heartbeat_back_addr":"172.21.15.61:6808/2536790504","heartbeat_front_addr":"172.21.15.61:6809/2536790504","state":["exists","up"]},{"osd":1,"uuid":"60ad73ed-e8f8-45eb-9628-28e316f03d77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":2624151341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6805","nonce":2624151341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":2624151341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6811","nonce":2624151341}]},"public_addr":"172.21.15.61:6804/2624151341","cluster_addr":"172.21.15.61:6805/2624151341","heartbeat_back_addr":"172.21.15.61:6810/2624151341","heartbeat_front_addr":"172.21.15.61:6811/2624151341","state":["exists","up"]},{"osd":2,"uuid":"756e2aa5-706f-43b3-9645-d8fd314d24ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":3069982037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6807","nonce":3069982037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":3069982037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6813","nonce":3069982037}]},"public_addr":"172.21.15.61:6806/3069982037","cluster_addr":"172.21.15.61:6807/3069982037","heartbeat_back_addr":"172.21.15.61:6812/3069982037","heartbeat_front_addr":"172.21.15.61:6813/3069982037","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-31T12:27:50.095 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-31T12:27:50.096 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-31T12:27:50.208 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:50.203+0000 7f5ffcf9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:50.210 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:50.208+0000 7f5ffcf9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:50.395 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:50.395 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":12,"fsid":"30686b99-d4b7-4afe-9de4-18fff3a2aa8e","created":"2023-10-31T12:27:37.848465+0000","modified":"2023-10-31T12:27:50.382181+0000","last_up_change":"2023-10-31T12:27:45.358885+0000","last_in_change":"2023-10-31T12:27:39.741843+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-31T12:27:46.410809+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-31T12:27:47.689691+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfb39abb-ee2e-48ae-b526-9bf48052c2f4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6803","nonce":2536790504}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":2536790504}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":2536790504}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6809","nonce":2536790504}]},"public_addr":"172.21.15.61:6803/2536790504","cluster_addr":"172.21.15.61:6802/2536790504","heartbeat_back_addr":"172.21.15.61:6808/2536790504","heartbeat_front_addr":"172.21.15.61:6809/2536790504","state":["exists","up"]},{"osd":1,"uuid":"60ad73ed-e8f8-45eb-9628-28e316f03d77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":2624151341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6805","nonce":2624151341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":2624151341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6811","nonce":2624151341}]},"public_addr":"172.21.15.61:6804/2624151341","cluster_addr":"172.21.15.61:6805/2624151341","heartbeat_back_addr":"172.21.15.61:6810/2624151341","heartbeat_front_addr":"172.21.15.61:6811/2624151341","state":["exists","up"]},{"osd":2,"uuid":"756e2aa5-706f-43b3-9645-d8fd314d24ad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":3069982037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6807","nonce":3069982037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":3069982037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6813","nonce":3069982037}]},"public_addr":"172.21.15.61:6806/3069982037","cluster_addr":"172.21.15.61:6807/3069982037","heartbeat_back_addr":"172.21.15.61:6812/3069982037","heartbeat_front_addr":"172.21.15.61:6813/3069982037","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-31T12:27:50.408 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-31T12:27:50.409 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-31T12:27:50.409 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-31T12:27:50.530 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:50.524+0000 7fd65f86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:50.531 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:50.524+0000 7f794759e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:50.531 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:50.524+0000 7f33f231e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:50.532 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:50.530+0000 7fd65f86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:50.533 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:50.530+0000 7f794759e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:50.533 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:50.530+0000 7f33f231e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:50.555 INFO:teuthology.orchestra.run.smithi061.stdout:34359738373 2023-10-31T12:27:50.555 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-31T12:27:50.559 INFO:teuthology.orchestra.run.smithi061.stdout:34359738373 2023-10-31T12:27:50.559 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-31T12:27:50.563 INFO:teuthology.orchestra.run.smithi061.stdout:34359738373 2023-10-31T12:27:50.564 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-31T12:27:50.670 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:50.664+0000 7f9a7e31e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:50.672 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:50.670+0000 7f9a7e31e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:50.721 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:50.714+0000 7f766b06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:50.721 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:50.714+0000 7fd05b55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:50.723 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:50.721+0000 7fd05b55e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:50.723 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:50.721+0000 7f766b06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:50.869 INFO:teuthology.orchestra.run.smithi061.stdout:34359738373 2023-10-31T12:27:50.881 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2023-10-31T12:27:50.882 DEBUG:teuthology.parallel:result is None 2023-10-31T12:27:50.912 INFO:teuthology.orchestra.run.smithi061.stdout:34359738373 2023-10-31T12:27:50.924 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2023-10-31T12:27:50.925 DEBUG:teuthology.parallel:result is None 2023-10-31T12:27:50.927 INFO:teuthology.orchestra.run.smithi061.stdout:34359738373 2023-10-31T12:27:50.939 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2023-10-31T12:27:50.940 DEBUG:teuthology.parallel:result is None 2023-10-31T12:27:50.940 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-31T12:27:50.940 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-31T12:27:51.091 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:51.086+0000 7f2cc3aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:51.093 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:51.091+0000 7f2cc3aad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:51.279 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:51.279 INFO:teuthology.orchestra.run.smithi061.stderr:dumped all 2023-10-31T12:27:51.290 INFO:teuthology.orchestra.run.smithi061.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-10-31T12:27:50.550534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80764,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034756,"statfs":{"total":289910292480,"available":289827590144,"internally_reserved":0,"allocated":274432,"data_stored":1227066,"data_compressed":59918,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-31T12:27:50.431911+0000","last_change":"2023-10-31T12:27:49.518842+0000","last_active":"2023-10-31T12:27:50.431911+0000","last_peered":"2023-10-31T12:27:50.431911+0000","last_clean":"2023-10-31T12:27:50.431911+0000","last_became_active":"2023-10-31T12:27:49.516118+0000","last_became_peered":"2023-10-31T12:27:49.516118+0000","last_unstale":"2023-10-31T12:27:50.431911+0000","last_undegraded":"2023-10-31T12:27:50.431911+0000","last_fullsized":"2023-10-31T12:27:50.431911+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_clean_scrub_stamp":"2023-10-31T12:27:48.372078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-31T12:27:50.432735+0000","last_change":"2023-10-31T12:27:49.532658+0000","last_active":"2023-10-31T12:27:50.432735+0000","last_peered":"2023-10-31T12:27:50.432735+0000","last_clean":"2023-10-31T12:27:50.432735+0000","last_became_active":"2023-10-31T12:27:49.530385+0000","last_became_peered":"2023-10-31T12:27:49.530385+0000","last_unstale":"2023-10-31T12:27:50.432735+0000","last_undegraded":"2023-10-31T12:27:50.432735+0000","last_fullsized":"2023-10-31T12:27:50.432735+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_clean_scrub_stamp":"2023-10-31T12:27:48.372078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-31T12:27:50.431242+0000","last_change":"2023-10-31T12:27:49.474200+0000","last_active":"2023-10-31T12:27:50.431242+0000","last_peered":"2023-10-31T12:27:50.431242+0000","last_clean":"2023-10-31T12:27:50.431242+0000","last_became_active":"2023-10-31T12:27:49.463037+0000","last_became_peered":"2023-10-31T12:27:49.463037+0000","last_unstale":"2023-10-31T12:27:50.431242+0000","last_undegraded":"2023-10-31T12:27:50.431242+0000","last_fullsized":"2023-10-31T12:27:50.431242+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_clean_scrub_stamp":"2023-10-31T12:27:48.372078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-31T12:27:50.435117+0000","last_change":"2023-10-31T12:27:49.511522+0000","last_active":"2023-10-31T12:27:50.435117+0000","last_peered":"2023-10-31T12:27:50.435117+0000","last_clean":"2023-10-31T12:27:50.435117+0000","last_became_active":"2023-10-31T12:27:49.509686+0000","last_became_peered":"2023-10-31T12:27:49.509686+0000","last_unstale":"2023-10-31T12:27:50.435117+0000","last_undegraded":"2023-10-31T12:27:50.435117+0000","last_fullsized":"2023-10-31T12:27:50.435117+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_clean_scrub_stamp":"2023-10-31T12:27:48.372078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-31T12:27:50.422892+0000","last_change":"2023-10-31T12:27:49.491458+0000","last_active":"2023-10-31T12:27:50.422892+0000","last_peered":"2023-10-31T12:27:50.422892+0000","last_clean":"2023-10-31T12:27:50.422892+0000","last_became_active":"2023-10-31T12:27:49.488376+0000","last_became_peered":"2023-10-31T12:27:49.488376+0000","last_unstale":"2023-10-31T12:27:50.422892+0000","last_undegraded":"2023-10-31T12:27:50.422892+0000","last_fullsized":"2023-10-31T12:27:50.422892+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_clean_scrub_stamp":"2023-10-31T12:27:48.372078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-31T12:27:50.391393+0000","last_change":"2023-10-31T12:27:49.496433+0000","last_active":"2023-10-31T12:27:50.391393+0000","last_peered":"2023-10-31T12:27:50.391393+0000","last_clean":"2023-10-31T12:27:50.391393+0000","last_became_active":"2023-10-31T12:27:49.493487+0000","last_became_peered":"2023-10-31T12:27:49.493487+0000","last_unstale":"2023-10-31T12:27:50.391393+0000","last_undegraded":"2023-10-31T12:27:50.391393+0000","last_fullsized":"2023-10-31T12:27:50.391393+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_clean_scrub_stamp":"2023-10-31T12:27:48.372078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-31T12:27:50.390302+0000","last_change":"2023-10-31T12:27:49.475771+0000","last_active":"2023-10-31T12:27:50.390302+0000","last_peered":"2023-10-31T12:27:50.390302+0000","last_clean":"2023-10-31T12:27:50.390302+0000","last_became_active":"2023-10-31T12:27:49.473409+0000","last_became_peered":"2023-10-31T12:27:49.473409+0000","last_unstale":"2023-10-31T12:27:50.390302+0000","last_undegraded":"2023-10-31T12:27:50.390302+0000","last_fullsized":"2023-10-31T12:27:50.390302+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_clean_scrub_stamp":"2023-10-31T12:27:48.372078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-31T12:27:50.430072+0000","last_change":"2023-10-31T12:27:49.486021+0000","last_active":"2023-10-31T12:27:50.430072+0000","last_peered":"2023-10-31T12:27:50.430072+0000","last_clean":"2023-10-31T12:27:50.430072+0000","last_became_active":"2023-10-31T12:27:49.483166+0000","last_became_peered":"2023-10-31T12:27:49.483166+0000","last_unstale":"2023-10-31T12:27:50.430072+0000","last_undegraded":"2023-10-31T12:27:50.430072+0000","last_fullsized":"2023-10-31T12:27:50.430072+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_clean_scrub_stamp":"2023-10-31T12:27:48.372078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-31T12:27:50.428606+0000","last_change":"2023-10-31T12:27:48.443033+0000","last_active":"2023-10-31T12:27:50.428606+0000","last_peered":"2023-10-31T12:27:50.428606+0000","last_clean":"2023-10-31T12:27:50.428606+0000","last_became_active":"2023-10-31T12:27:48.437912+0000","last_became_peered":"2023-10-31T12:27:48.437912+0000","last_unstale":"2023-10-31T12:27:50.428606+0000","last_undegraded":"2023-10-31T12:27:50.428606+0000","last_fullsized":"2023-10-31T12:27:50.428606+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:27:47.363954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:27:47.363954+0000","last_clean_scrub_stamp":"2023-10-31T12:27:47.363954+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":16984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-10-31T12:27:51.291 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-31T12:27:51.404 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:51.399+0000 7fd4e9d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:51.406 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:51.404+0000 7fd4e9d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:51.585 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:51.586 INFO:teuthology.orchestra.run.smithi061.stderr:dumped all 2023-10-31T12:27:51.597 INFO:teuthology.orchestra.run.smithi061.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2023-10-31T12:27:50.550534+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80764,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034756,"statfs":{"total":289910292480,"available":289827590144,"internally_reserved":0,"allocated":274432,"data_stored":1227066,"data_compressed":59918,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-31T12:27:50.431911+0000","last_change":"2023-10-31T12:27:49.518842+0000","last_active":"2023-10-31T12:27:50.431911+0000","last_peered":"2023-10-31T12:27:50.431911+0000","last_clean":"2023-10-31T12:27:50.431911+0000","last_became_active":"2023-10-31T12:27:49.516118+0000","last_became_peered":"2023-10-31T12:27:49.516118+0000","last_unstale":"2023-10-31T12:27:50.431911+0000","last_undegraded":"2023-10-31T12:27:50.431911+0000","last_fullsized":"2023-10-31T12:27:50.431911+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_clean_scrub_stamp":"2023-10-31T12:27:48.372078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-31T12:27:50.432735+0000","last_change":"2023-10-31T12:27:49.532658+0000","last_active":"2023-10-31T12:27:50.432735+0000","last_peered":"2023-10-31T12:27:50.432735+0000","last_clean":"2023-10-31T12:27:50.432735+0000","last_became_active":"2023-10-31T12:27:49.530385+0000","last_became_peered":"2023-10-31T12:27:49.530385+0000","last_unstale":"2023-10-31T12:27:50.432735+0000","last_undegraded":"2023-10-31T12:27:50.432735+0000","last_fullsized":"2023-10-31T12:27:50.432735+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_clean_scrub_stamp":"2023-10-31T12:27:48.372078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-31T12:27:50.431242+0000","last_change":"2023-10-31T12:27:49.474200+0000","last_active":"2023-10-31T12:27:50.431242+0000","last_peered":"2023-10-31T12:27:50.431242+0000","last_clean":"2023-10-31T12:27:50.431242+0000","last_became_active":"2023-10-31T12:27:49.463037+0000","last_became_peered":"2023-10-31T12:27:49.463037+0000","last_unstale":"2023-10-31T12:27:50.431242+0000","last_undegraded":"2023-10-31T12:27:50.431242+0000","last_fullsized":"2023-10-31T12:27:50.431242+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_clean_scrub_stamp":"2023-10-31T12:27:48.372078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-31T12:27:50.435117+0000","last_change":"2023-10-31T12:27:49.511522+0000","last_active":"2023-10-31T12:27:50.435117+0000","last_peered":"2023-10-31T12:27:50.435117+0000","last_clean":"2023-10-31T12:27:50.435117+0000","last_became_active":"2023-10-31T12:27:49.509686+0000","last_became_peered":"2023-10-31T12:27:49.509686+0000","last_unstale":"2023-10-31T12:27:50.435117+0000","last_undegraded":"2023-10-31T12:27:50.435117+0000","last_fullsized":"2023-10-31T12:27:50.435117+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_clean_scrub_stamp":"2023-10-31T12:27:48.372078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-31T12:27:50.422892+0000","last_change":"2023-10-31T12:27:49.491458+0000","last_active":"2023-10-31T12:27:50.422892+0000","last_peered":"2023-10-31T12:27:50.422892+0000","last_clean":"2023-10-31T12:27:50.422892+0000","last_became_active":"2023-10-31T12:27:49.488376+0000","last_became_peered":"2023-10-31T12:27:49.488376+0000","last_unstale":"2023-10-31T12:27:50.422892+0000","last_undegraded":"2023-10-31T12:27:50.422892+0000","last_fullsized":"2023-10-31T12:27:50.422892+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_clean_scrub_stamp":"2023-10-31T12:27:48.372078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-31T12:27:50.391393+0000","last_change":"2023-10-31T12:27:49.496433+0000","last_active":"2023-10-31T12:27:50.391393+0000","last_peered":"2023-10-31T12:27:50.391393+0000","last_clean":"2023-10-31T12:27:50.391393+0000","last_became_active":"2023-10-31T12:27:49.493487+0000","last_became_peered":"2023-10-31T12:27:49.493487+0000","last_unstale":"2023-10-31T12:27:50.391393+0000","last_undegraded":"2023-10-31T12:27:50.391393+0000","last_fullsized":"2023-10-31T12:27:50.391393+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_clean_scrub_stamp":"2023-10-31T12:27:48.372078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-31T12:27:50.390302+0000","last_change":"2023-10-31T12:27:49.475771+0000","last_active":"2023-10-31T12:27:50.390302+0000","last_peered":"2023-10-31T12:27:50.390302+0000","last_clean":"2023-10-31T12:27:50.390302+0000","last_became_active":"2023-10-31T12:27:49.473409+0000","last_became_peered":"2023-10-31T12:27:49.473409+0000","last_unstale":"2023-10-31T12:27:50.390302+0000","last_undegraded":"2023-10-31T12:27:50.390302+0000","last_fullsized":"2023-10-31T12:27:50.390302+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_clean_scrub_stamp":"2023-10-31T12:27:48.372078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-31T12:27:50.430072+0000","last_change":"2023-10-31T12:27:49.486021+0000","last_active":"2023-10-31T12:27:50.430072+0000","last_peered":"2023-10-31T12:27:50.430072+0000","last_clean":"2023-10-31T12:27:50.430072+0000","last_became_active":"2023-10-31T12:27:49.483166+0000","last_became_peered":"2023-10-31T12:27:49.483166+0000","last_unstale":"2023-10-31T12:27:50.430072+0000","last_undegraded":"2023-10-31T12:27:50.430072+0000","last_fullsized":"2023-10-31T12:27:50.430072+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:27:48.372078+0000","last_clean_scrub_stamp":"2023-10-31T12:27:48.372078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-31T12:27:50.428606+0000","last_change":"2023-10-31T12:27:48.443033+0000","last_active":"2023-10-31T12:27:50.428606+0000","last_peered":"2023-10-31T12:27:50.428606+0000","last_clean":"2023-10-31T12:27:50.428606+0000","last_became_active":"2023-10-31T12:27:48.437912+0000","last_became_peered":"2023-10-31T12:27:48.437912+0000","last_unstale":"2023-10-31T12:27:50.428606+0000","last_undegraded":"2023-10-31T12:27:50.428606+0000","last_fullsized":"2023-10-31T12:27:50.428606+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-31T12:27:47.363954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-31T12:27:47.363954+0000","last_clean_scrub_stamp":"2023-10-31T12:27:47.363954+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":16984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29959,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-10-31T12:27:51.598 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-31T12:27:51.598 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-31T12:27:51.598 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-31T12:27:51.599 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-31T12:27:51.708 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:51.704+0000 7f88c986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:51.710 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:27:51.708+0000 7f88c986d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:27:51.934 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:51.934 INFO:teuthology.orchestra.run.smithi061.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-31T12:27:51.946 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-31T12:27:51.947 INFO:teuthology.run_tasks:Running task cbt... 2023-10-31T12:27:51.959 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi061.front.sepia.ceph.com', 'osds': ['ubuntu@smithi061.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi061.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi061.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi061.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4096], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2023-10-31T12:27:51.960 DEBUG:teuthology.orchestra.run.smithi061:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2023-10-31T12:27:51.978 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:27:51.978 DEBUG:teuthology.orchestra.run.smithi061:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-10-31T12:27:52.031 DEBUG:teuthology.orchestra.run.smithi061:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2023-10-31T12:27:52.086 INFO:teuthology.orchestra.run.smithi061.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2023-10-31T12:27:52.484 INFO:tasks.cbt:Installing collectl 2023-10-31T12:27:52.485 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2023-10-31T12:27:52.615 INFO:teuthology.orchestra.run.smithi061.stderr:--2023-10-31 12:27:52-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2023-10-31T12:27:52.750 INFO:teuthology.orchestra.run.smithi061.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::6812:256f, ... 2023-10-31T12:27:52.760 INFO:teuthology.orchestra.run.smithi061.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected. 2023-10-31T12:27:52.994 INFO:teuthology.orchestra.run.smithi061.stderr:HTTP request sent, awaiting response... 302 Found 2023-10-31T12:27:52.995 INFO:teuthology.orchestra.run.smithi061.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlQPLIBPxsZE3V-knUfH0JVhNoxC6pXHpkCTNL4FdNRS9JLbp0PeqPeZcEBrBof5Z2i_yP4Ur9xyCYOlHGtnWKYyMNCg%3D%3D&use_mirror=newcontinuum&r= [following] 2023-10-31T12:27:52.995 INFO:teuthology.orchestra.run.smithi061.stderr:--2023-10-31 12:27:52-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlQPLIBPxsZE3V-knUfH0JVhNoxC6pXHpkCTNL4FdNRS9JLbp0PeqPeZcEBrBof5Z2i_yP4Ur9xyCYOlHGtnWKYyMNCg%3D%3D&use_mirror=newcontinuum&r= 2023-10-31T12:27:53.006 INFO:teuthology.orchestra.run.smithi061.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2023-10-31T12:27:53.078 INFO:teuthology.orchestra.run.smithi061.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2023-10-31T12:27:53.331 INFO:teuthology.orchestra.run.smithi061.stderr:HTTP request sent, awaiting response... 302 Found 2023-10-31T12:27:53.332 INFO:teuthology.orchestra.run.smithi061.stderr:Location: https://newcontinuum.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2023-10-31T12:27:53.332 INFO:teuthology.orchestra.run.smithi061.stderr:--2023-10-31 12:27:53-- https://newcontinuum.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2023-10-31T12:27:53.332 INFO:teuthology.orchestra.run.smithi061.stderr:Resolving newcontinuum.dl.sourceforge.net (newcontinuum.dl.sourceforge.net)... 64.79.96.4, 2607:ff50:0:11::32 2023-10-31T12:27:53.362 INFO:teuthology.orchestra.run.smithi061.stderr:Connecting to newcontinuum.dl.sourceforge.net (newcontinuum.dl.sourceforge.net)|64.79.96.4|:443... connected. 2023-10-31T12:27:53.462 INFO:teuthology.orchestra.run.smithi061.stderr:HTTP request sent, awaiting response... 200 OK 2023-10-31T12:27:53.462 INFO:teuthology.orchestra.run.smithi061.stderr:Length: 620693 (606K) [application/x-gzip] 2023-10-31T12:27:53.463 INFO:teuthology.orchestra.run.smithi061.stderr:Saving to: ‘collectl.tar.gz’ 2023-10-31T12:27:53.492 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-10-31T12:27:53.522 INFO:teuthology.orchestra.run.smithi061.stderr: 0K .......... .......... .......... .......... .......... 8% 828K 1s 2023-10-31T12:27:53.551 INFO:teuthology.orchestra.run.smithi061.stderr: 50K .......... .......... .......... .......... .......... 16% 1.69M 0s 2023-10-31T12:27:53.553 INFO:teuthology.orchestra.run.smithi061.stderr: 100K .......... .......... .......... .......... .......... 24% 28.8M 0s 2023-10-31T12:27:53.580 INFO:teuthology.orchestra.run.smithi061.stderr: 150K .......... .......... .......... .......... .......... 32% 1.82M 0s 2023-10-31T12:27:53.582 INFO:teuthology.orchestra.run.smithi061.stderr: 200K .......... .......... .......... .......... .......... 41% 21.9M 0s 2023-10-31T12:27:53.585 INFO:teuthology.orchestra.run.smithi061.stderr: 250K .......... .......... .......... .......... .......... 49% 21.1M 0s 2023-10-31T12:27:53.586 INFO:teuthology.orchestra.run.smithi061.stderr: 300K .......... .......... .......... .......... .......... 57% 23.5M 0s 2023-10-31T12:27:53.608 INFO:teuthology.orchestra.run.smithi061.stderr: 350K .......... .......... .......... .......... .......... 65% 2.33M 0s 2023-10-31T12:27:53.610 INFO:teuthology.orchestra.run.smithi061.stderr: 400K .......... .......... .......... .......... .......... 74% 15.8M 0s 2023-10-31T12:27:53.612 INFO:teuthology.orchestra.run.smithi061.stderr: 450K .......... .......... .......... .......... .......... 82% 42.6M 0s 2023-10-31T12:27:53.614 INFO:teuthology.orchestra.run.smithi061.stderr: 500K .......... .......... .......... .......... .......... 90% 20.9M 0s 2023-10-31T12:27:53.616 INFO:teuthology.orchestra.run.smithi061.stderr: 550K .......... .......... .......... .......... .......... 98% 24.7M 0s 2023-10-31T12:27:53.616 INFO:teuthology.orchestra.run.smithi061.stderr: 600K ...... 100% 228M=0.2s 2023-10-31T12:27:53.616 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-10-31T12:27:53.617 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31 12:27:53 (3.85 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2023-10-31T12:27:53.617 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-10-31T12:27:53.654 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/ 2023-10-31T12:27:53.655 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/collectl.conf 2023-10-31T12:27:53.655 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/ 2023-10-31T12:27:53.655 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/SlowProc.html 2023-10-31T12:27:53.656 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/BuddyInfo.html 2023-10-31T12:27:53.656 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/WhySummary.html 2023-10-31T12:27:53.656 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Socket.html 2023-10-31T12:27:53.656 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/colmux.html 2023-10-31T12:27:53.656 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Tutorial.html 2023-10-31T12:27:53.656 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Gexpr.html 2023-10-31T12:27:53.657 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Import.html 2023-10-31T12:27:53.657 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/style.css 2023-10-31T12:27:53.657 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2023-10-31T12:27:53.657 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2023-10-31T12:27:53.657 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/FileNaming.html 2023-10-31T12:27:53.658 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/TheMath.html 2023-10-31T12:27:53.658 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Environmental.html 2023-10-31T12:27:53.658 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Process.html 2023-10-31T12:27:53.658 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/OutputFiles.html 2023-10-31T12:27:53.658 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/HiResTime.html 2023-10-31T12:27:53.659 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/OutputFormats.html 2023-10-31T12:27:53.659 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/slub.jpg 2023-10-31T12:27:53.659 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Lustre.html 2023-10-31T12:27:53.659 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Documentation.html 2023-10-31T12:27:53.659 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Inodes.html 2023-10-31T12:27:53.660 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Graphite.html 2023-10-31T12:27:53.660 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/InputFiles.html 2023-10-31T12:27:53.660 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Export.html 2023-10-31T12:27:53.660 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Interrupts.html 2023-10-31T12:27:53.660 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Startup.html 2023-10-31T12:27:53.661 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Performance.html 2023-10-31T12:27:53.661 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Messages.html 2023-10-31T12:27:53.661 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Logging.html 2023-10-31T12:27:53.661 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Data-detail.html 2023-10-31T12:27:53.661 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/CPUs.html 2023-10-31T12:27:53.662 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Matrix.html 2023-10-31T12:27:53.662 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Data-brief.html 2023-10-31T12:27:53.662 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/SlabInfo.html 2023-10-31T12:27:53.662 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Misc.html 2023-10-31T12:27:53.662 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Examples.html 2023-10-31T12:27:53.663 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/OpenStack.html 2023-10-31T12:27:53.663 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Plotfiles.html 2023-10-31T12:27:53.663 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2023-10-31T12:27:53.663 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2023-10-31T12:27:53.663 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Data-verbose.html 2023-10-31T12:27:53.664 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Data.html 2023-10-31T12:27:53.664 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Exceptions.html 2023-10-31T12:27:53.664 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/RunningAsAService.html 2023-10-31T12:27:53.664 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2023-10-31T12:27:53.664 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Ganglia.jpg 2023-10-31T12:27:53.665 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/OperationalModes.html 2023-10-31T12:27:53.665 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Playback.html 2023-10-31T12:27:53.665 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Hello.html 2023-10-31T12:27:53.665 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Sockets.html 2023-10-31T12:27:53.665 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Network.html 2023-10-31T12:27:53.666 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Infiniband.html 2023-10-31T12:27:53.666 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Memory.html 2023-10-31T12:27:53.666 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2023-10-31T12:27:53.666 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/Disks.html 2023-10-31T12:27:53.666 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/docs/NfsInfo.html 2023-10-31T12:27:53.667 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/service/ 2023-10-31T12:27:53.667 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/service/collectl.service 2023-10-31T12:27:53.667 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/initd/ 2023-10-31T12:27:53.667 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/initd/collectl-suse 2023-10-31T12:27:53.667 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/initd/collectl-debian 2023-10-31T12:27:53.668 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/initd/collectl-generic 2023-10-31T12:27:53.668 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/initd/collectl 2023-10-31T12:27:53.668 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/misc.ph 2023-10-31T12:27:53.668 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/gexpr.ph 2023-10-31T12:27:53.668 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/statsd.ph 2023-10-31T12:27:53.668 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/envrules.std 2023-10-31T12:27:53.669 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/hello.ph 2023-10-31T12:27:53.669 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/vmsum.ph 2023-10-31T12:27:53.669 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/proctree.ph 2023-10-31T12:27:53.669 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/README-WINDOWS 2023-10-31T12:27:53.670 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/formatit.ph 2023-10-31T12:27:53.670 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/colmux 2023-10-31T12:27:53.671 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/INSTALL 2023-10-31T12:27:53.672 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/graphite.ph 2023-10-31T12:27:53.672 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/RELEASE-collectl 2023-10-31T12:27:53.672 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/vmstat.ph 2023-10-31T12:27:53.672 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/README 2023-10-31T12:27:53.673 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/man1/ 2023-10-31T12:27:53.673 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/man1/collectl.1 2023-10-31T12:27:53.673 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/man1/colmux.1 2023-10-31T12:27:53.673 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/lexpr.ph 2023-10-31T12:27:53.673 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/client.pl 2023-10-31T12:27:53.674 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/collectl 2023-10-31T12:27:53.674 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/COPYING 2023-10-31T12:27:53.674 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/vnet.ph 2023-10-31T12:27:53.674 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/GPL 2023-10-31T12:27:53.675 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/UNINSTALL 2023-10-31T12:27:53.675 INFO:teuthology.orchestra.run.smithi061.stdout:collectl-4.3.1/ARTISTIC 2023-10-31T12:27:53.719 INFO:teuthology.orchestra.run.smithi061.stderr:./INSTALL: line 90: chkconfig: command not found 2023-10-31T12:27:53.720 DEBUG:teuthology.orchestra.run.smithi061:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2023-10-31T12:27:54.289 INFO:teuthology.orchestra.run.smithi061.stdout:Last metadata expiration check: 0:03:23 ago on Tue 31 Oct 2023 12:24:31 PM UTC. 2023-10-31T12:27:54.385 INFO:teuthology.orchestra.run.smithi061.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2023-10-31T12:27:54.388 INFO:teuthology.orchestra.run.smithi061.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2023-10-31T12:27:54.446 INFO:teuthology.orchestra.run.smithi061.stdout:Dependencies resolved. 2023-10-31T12:27:54.447 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:27:54.447 INFO:teuthology.orchestra.run.smithi061.stdout: Package Arch Version Repository Size 2023-10-31T12:27:54.447 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:27:54.447 INFO:teuthology.orchestra.run.smithi061.stdout:Installing: 2023-10-31T12:27:54.447 INFO:teuthology.orchestra.run.smithi061.stdout: librbd-devel x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 23 k 2023-10-31T12:27:54.448 INFO:teuthology.orchestra.run.smithi061.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2023-10-31T12:27:54.448 INFO:teuthology.orchestra.run.smithi061.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2023-10-31T12:27:54.448 INFO:teuthology.orchestra.run.smithi061.stdout: perf x86_64 5.14.0-378.el9 appstream 7.3 M 2023-10-31T12:27:54.448 INFO:teuthology.orchestra.run.smithi061.stdout:Installing dependencies: 2023-10-31T12:27:54.448 INFO:teuthology.orchestra.run.smithi061.stdout: libradospp-devel x86_64 2:18.0.0-6792.ge83bf482.el9 ceph 30 k 2023-10-31T12:27:54.449 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:54.449 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction Summary 2023-10-31T12:27:54.449 INFO:teuthology.orchestra.run.smithi061.stdout:================================================================================ 2023-10-31T12:27:54.449 INFO:teuthology.orchestra.run.smithi061.stdout:Install 5 Packages 2023-10-31T12:27:54.449 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:54.450 INFO:teuthology.orchestra.run.smithi061.stdout:Total download size: 7.5 M 2023-10-31T12:27:54.450 INFO:teuthology.orchestra.run.smithi061.stdout:Installed size: 11 M 2023-10-31T12:27:54.451 INFO:teuthology.orchestra.run.smithi061.stdout:Downloading Packages: 2023-10-31T12:27:54.855 INFO:teuthology.orchestra.run.smithi061.stdout:(1/5): libradospp-devel-18.0.0-6792.ge83bf482.e 176 kB/s | 30 kB 00:00 2023-10-31T12:27:54.880 INFO:teuthology.orchestra.run.smithi061.stdout:(2/5): librbd-devel-18.0.0-6792.ge83bf482.el9.x 118 kB/s | 23 kB 00:00 2023-10-31T12:27:54.955 INFO:teuthology.orchestra.run.smithi061.stdout:(3/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 175 kB/s | 13 kB 00:00 2023-10-31T12:27:55.072 INFO:teuthology.orchestra.run.smithi061.stdout:(4/5): perf-5.14.0-378.el9.x86_64.rpm 19 MB/s | 7.3 MB 00:00 2023-10-31T12:27:55.097 INFO:teuthology.orchestra.run.smithi061.stdout:(5/5): pdsh-2.34-7.el9.x86_64.rpm 489 kB/s | 118 kB 00:00 2023-10-31T12:27:55.098 INFO:teuthology.orchestra.run.smithi061.stdout:-------------------------------------------------------------------------------- 2023-10-31T12:27:55.099 INFO:teuthology.orchestra.run.smithi061.stdout:Total 12 MB/s | 7.5 MB 00:00 2023-10-31T12:27:55.178 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction check 2023-10-31T12:27:55.201 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction check succeeded. 2023-10-31T12:27:55.201 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction test 2023-10-31T12:27:55.454 INFO:teuthology.orchestra.run.smithi061.stdout:Transaction test succeeded. 2023-10-31T12:27:55.455 INFO:teuthology.orchestra.run.smithi061.stdout:Running transaction 2023-10-31T12:27:55.803 INFO:teuthology.orchestra.run.smithi061.stdout: Preparing : 1/1 2023-10-31T12:27:55.877 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2023-10-31T12:27:55.905 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2023-10-31T12:27:55.927 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : libradospp-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 3/5 2023-10-31T12:27:56.276 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : librbd-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 4/5 2023-10-31T12:27:56.395 INFO:teuthology.orchestra.run.smithi061.stdout: Installing : perf-5.14.0-378.el9.x86_64 5/5 2023-10-31T12:27:57.260 INFO:teuthology.orchestra.run.smithi061.stdout: Running scriptlet: perf-5.14.0-378.el9.x86_64 5/5 2023-10-31T12:27:57.260 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : libradospp-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 1/5 2023-10-31T12:27:57.260 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : librbd-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 2/5 2023-10-31T12:27:57.261 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : perf-5.14.0-378.el9.x86_64 3/5 2023-10-31T12:27:57.261 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2023-10-31T12:27:57.592 INFO:teuthology.orchestra.run.smithi061.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2023-10-31T12:27:57.593 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:57.593 INFO:teuthology.orchestra.run.smithi061.stdout:Installed: 2023-10-31T12:27:57.593 INFO:teuthology.orchestra.run.smithi061.stdout: libradospp-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:27:57.593 INFO:teuthology.orchestra.run.smithi061.stdout: librbd-devel-2:18.0.0-6792.ge83bf482.el9.x86_64 2023-10-31T12:27:57.593 INFO:teuthology.orchestra.run.smithi061.stdout: pdsh-2.34-7.el9.x86_64 2023-10-31T12:27:57.594 INFO:teuthology.orchestra.run.smithi061.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2023-10-31T12:27:57.594 INFO:teuthology.orchestra.run.smithi061.stdout: perf-5.14.0-378.el9.x86_64 2023-10-31T12:27:57.594 INFO:teuthology.orchestra.run.smithi061.stdout: 2023-10-31T12:27:57.594 INFO:teuthology.orchestra.run.smithi061.stdout:Complete! 2023-10-31T12:27:57.992 INFO:tasks.cbt:benchmark: radosbench 2023-10-31T12:27:57.993 DEBUG:teuthology.orchestra.run.smithi061:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2023-10-31T12:27:58.026 DEBUG:teuthology.orchestra.run.smithi061:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-10-31T12:27:58.461 INFO:teuthology.orchestra.run.smithi061.stderr:12:27:58 - DEBUG - cbt - Settings.cluster: 2023-10-31T12:27:58.461 INFO:teuthology.orchestra.run.smithi061.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2023-10-31T12:27:58.461 INFO:teuthology.orchestra.run.smithi061.stderr: 'clients': ['ubuntu@smithi061.front.sepia.ceph.com'], 2023-10-31T12:27:58.461 INFO:teuthology.orchestra.run.smithi061.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2023-10-31T12:27:58.462 INFO:teuthology.orchestra.run.smithi061.stderr: 'head': 'ubuntu@smithi061.front.sepia.ceph.com', 2023-10-31T12:27:58.462 INFO:teuthology.orchestra.run.smithi061.stderr: 'is_teuthology': True, 2023-10-31T12:27:58.462 INFO:teuthology.orchestra.run.smithi061.stderr: 'iterations': 1, 2023-10-31T12:27:58.462 INFO:teuthology.orchestra.run.smithi061.stderr: 'mons': ['ubuntu@smithi061.front.sepia.ceph.com'], 2023-10-31T12:27:58.463 INFO:teuthology.orchestra.run.smithi061.stderr: 'osds': ['ubuntu@smithi061.front.sepia.ceph.com'], 2023-10-31T12:27:58.463 INFO:teuthology.orchestra.run.smithi061.stderr: 'osds_per_node': 3, 2023-10-31T12:27:58.463 INFO:teuthology.orchestra.run.smithi061.stderr: 'pid_dir': '/var/run/ceph', 2023-10-31T12:27:58.463 INFO:teuthology.orchestra.run.smithi061.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2023-10-31T12:27:58.463 INFO:teuthology.orchestra.run.smithi061.stderr: 'pgp_size': 256, 2023-10-31T12:27:58.463 INFO:teuthology.orchestra.run.smithi061.stderr: 'replication': 'replicated'}}, 2023-10-31T12:27:58.464 INFO:teuthology.orchestra.run.smithi061.stderr: 'rebuild_every_test': False, 2023-10-31T12:27:58.464 INFO:teuthology.orchestra.run.smithi061.stderr: 'rgws': {'ubuntu@smithi061.front.sepia.ceph.com': None}, 2023-10-31T12:27:58.464 INFO:teuthology.orchestra.run.smithi061.stderr: 'tmp_dir': '/tmp/cbt', 2023-10-31T12:27:58.464 INFO:teuthology.orchestra.run.smithi061.stderr: 'use_existing': True, 2023-10-31T12:27:58.464 INFO:teuthology.orchestra.run.smithi061.stderr: 'user': 'ubuntu'} 2023-10-31T12:27:58.482 INFO:teuthology.orchestra.run.smithi061.stderr:12:27:58 - INFO - cbt - Pausing for 60s for idle monitoring. 2023-10-31T12:27:58.484 INFO:teuthology.orchestra.run.smithi061.stderr:12:27:58 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:27:58.484 INFO:teuthology.orchestra.run.smithi061.stderr:12:27:58 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:27:58.484 INFO:teuthology.orchestra.run.smithi061.stderr:12:27:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl 2023-10-31T12:27:58.487 INFO:teuthology.orchestra.run.smithi061.stderr:12:27:58 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:27:58.487 INFO:teuthology.orchestra.run.smithi061.stderr:12:27:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-10-31T12:27:58.488 INFO:teuthology.orchestra.run.smithi061.stderr:12:27:58 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:27:58.489 INFO:teuthology.orchestra.run.smithi061.stderr:12:27:58 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:27:58.489 INFO:teuthology.orchestra.run.smithi061.stderr:12:27:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf 2023-10-31T12:27:58.491 INFO:teuthology.orchestra.run.smithi061.stderr:12:27:58 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:27:58.492 INFO:teuthology.orchestra.run.smithi061.stderr:12:27:58 - DEBUG - cbt - PerfMonitoring: in local_node 2023-10-31T12:27:58.492 INFO:teuthology.orchestra.run.smithi061.stderr:12:27:58 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-10-31T12:28:58.524 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:28:58.524 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-10-31T12:28:58.567 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:28:58.567 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-10-31T12:28:58.613 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:28:58.613 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2023-10-31T12:28:58.615 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:28:58.615 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2023-10-31T12:28:58.617 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:28:58.617 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:28:58.617 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/* 2023-10-31T12:28:58.637 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:28:58.637 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/$(basename ${i}).smithi061.front.sepia.ceph.com ; done 2023-10-31T12:28:58.641 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:28:58.641 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:28:58.642 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo killall -9 rados 2023-10-31T12:28:58.666 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - WARNING - cbt - sudo killall -9 rados 2023-10-31T12:28:58.666 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - WARNING - cbt - error 1 seen, continuing anyway... 2023-10-31T12:28:58.686 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af. 2023-10-31T12:28:58.686 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004 2023-10-31T12:28:58.686 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:28:58.687 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:28:58.687 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004 2023-10-31T12:28:58.705 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:28:58.705 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:28:58.705 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl 2023-10-31T12:28:58.708 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:28:58.708 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-10-31T12:28:58.709 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:28:58.710 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:28:58.710 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf 2023-10-31T12:28:58.713 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:28:58.713 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - PerfMonitoring: in local_node 2023-10-31T12:28:58.713 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-10-31T12:28:58.713 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:28:58.713 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:28:58.714 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:58 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2023-10-31T12:28:59.018 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:59 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:28:59.018 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:59 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:28:59.019 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:59 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2023-10-31T12:28:59.904 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:59 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:28:59.905 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:59 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:28:59.905 INFO:teuthology.orchestra.run.smithi061.stderr:12:28:59 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2023-10-31T12:29:00.913 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:00 - INFO - cbt - Checking Health after pool creation. 2023-10-31T12:29:05.919 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:05 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:29:05.919 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:05 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:29:05.919 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-10-31T12:29:06.235 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:06 - INFO - cbt - Final Pool Health Check. 2023-10-31T12:29:11.236 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:11 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:29:11.236 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:11 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:29:11.236 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-10-31T12:29:11.563 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:11 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:29:11.563 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-10-31T12:29:11.587 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:11 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:29:11.587 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-10-31T12:29:11.626 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:11 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:29:11.627 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2023-10-31T12:29:11.628 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:11 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:29:11.628 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2023-10-31T12:29:11.630 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:11 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:29:11.630 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:11 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:29:11.630 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2023-10-31T12:29:11.791 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:11 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:29:11.792 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:11 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2023-10-31T12:29:12.351 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:12 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:29:12.352 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:12 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:29:12.352 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2023-10-31T12:29:12.645 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:12 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:29:12.646 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:12 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:29:12.646 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:12 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write 2023-10-31T12:29:12.716 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:12 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:29:12.716 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:12 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:29:12.716 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:12 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/ceph_settings.out 2023-10-31T12:29:13.497 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:13 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:29:13.497 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:13 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:29:13.497 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/collectl 2023-10-31T12:29:13.499 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:13 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:29:13.500 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-10-31T12:29:13.501 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:13 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:29:13.501 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:13 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:29:13.502 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf 2023-10-31T12:29:13.504 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:13 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:29:13.505 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:13 - DEBUG - cbt - PerfMonitoring: in local_node 2023-10-31T12:29:13.505 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:13 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-10-31T12:29:13.505 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:13 - INFO - cbt - Running radosbench write test. 2023-10-31T12:29:13.506 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:13 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:29:13.506 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:13 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:29:13.506 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/output.0 2023-10-31T12:29:13.507 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:13 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:29:13.507 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:13 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:29:13.507 INFO:teuthology.orchestra.run.smithi061.stderr:12:29:13 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4096 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/output.1 2023-10-31T12:30:14.615 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:14 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:14.615 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-10-31T12:30:14.646 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:14 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:14.647 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-10-31T12:30:14.688 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:14 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:14.688 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf/perf.data 2023-10-31T12:30:14.690 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:14 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:14.690 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/perf/perf_stat.* 2023-10-31T12:30:14.692 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:14 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:30:14.692 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:14 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:14.693 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/historic_ops.out 2023-10-31T12:30:15.025 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:15 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:30:15.041 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:15 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:15.042 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:15 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/* 2023-10-31T12:30:15.060 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:15 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:15.061 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:15 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/write/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/write/$(basename ${i}).smithi061.front.sepia.ceph.com ; done 2023-10-31T12:30:15.090 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:15 - INFO - cbt - Convert results to json format. 2023-10-31T12:30:15.090 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:15 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:30:15.091 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:15 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:30:15.091 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:15 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:15.091 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2023-10-31T12:30:15.953 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:15 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:15.953 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2023-10-31T12:30:16.210 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:16 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:30:16.211 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:16 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:16.211 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:16 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2023-10-31T12:30:16.520 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:16 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:30:16.521 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:16 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:16.521 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:16 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/seq 2023-10-31T12:30:16.567 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:16 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:30:16.568 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:16 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:16.568 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:16 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/seq/ceph_settings.out 2023-10-31T12:30:17.289 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:17 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:30:17.289 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:17 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:17.289 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:17 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/seq/collectl 2023-10-31T12:30:17.291 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:17 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:17.292 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:17 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/seq/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-10-31T12:30:17.293 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:17 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:30:17.293 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:17 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:17.293 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:17 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/seq/perf 2023-10-31T12:30:17.296 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:17 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:17.297 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:17 - DEBUG - cbt - PerfMonitoring: in local_node 2023-10-31T12:30:17.297 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:17 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-10-31T12:30:17.297 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:17 - INFO - cbt - Running radosbench seq test. 2023-10-31T12:30:17.297 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:17 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:30:17.298 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:17 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:17.298 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:17 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 seq --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/seq/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/seq/output.0 2023-10-31T12:30:17.298 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:17 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:30:17.299 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:17 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:17.299 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:17 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench 60 seq --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/seq/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/seq/output.1 2023-10-31T12:30:30.440 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:30 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:30.440 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-10-31T12:30:30.471 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:30 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:30.471 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-10-31T12:30:30.513 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:30 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:30.513 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/seq/perf/perf.data 2023-10-31T12:30:30.515 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:30 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:30.515 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/seq/perf/perf_stat.* 2023-10-31T12:30:30.517 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:30 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:30:30.517 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:30 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:30.517 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:30 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/seq/historic_ops.out 2023-10-31T12:30:30.879 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:30 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:30:30.882 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:30 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:30.882 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:30 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/seq/* 2023-10-31T12:30:30.903 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:30 - DEBUG - cbt - get_fqdn_local()=smithi061.front.sepia.ceph.com 2023-10-31T12:30:30.904 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:30 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-00004096/concurrent_ops-00000004/seq/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/seq/$(basename ${i}).smithi061.front.sepia.ceph.com ; done 2023-10-31T12:30:30.930 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:30 - INFO - cbt - Convert results to json format. 2023-10-31T12:30:30.930 INFO:teuthology.orchestra.run.smithi061.stderr:12:30:30 - DEBUG - cbt - Nodes : ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:30:30.949 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2023-10-31T12:30:30.958 DEBUG:teuthology.orchestra.run.smithi061:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2023-10-31T12:30:31.130 INFO:tasks.cbt_performance:Collecting CBT performance data config 2023-10-31T12:30:31.131 INFO:tasks.cbt_performance:reading cbt preferences 2023-10-31T12:30:31.132 DEBUG:teuthology.orchestra.run.smithi061:> find /home/ubuntu/cephtest -name 'json_output.*' 2023-10-31T12:30:31.187 INFO:teuthology.orchestra.run.smithi061.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/write/json_output.0.smithi061.front.sepia.ceph.com 2023-10-31T12:30:31.187 INFO:teuthology.orchestra.run.smithi061.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/write/json_output.1.smithi061.front.sepia.ceph.com 2023-10-31T12:30:31.187 INFO:teuthology.orchestra.run.smithi061.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/seq/json_output.0.smithi061.front.sepia.ceph.com 2023-10-31T12:30:31.188 INFO:teuthology.orchestra.run.smithi061.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/seq/json_output.1.smithi061.front.sepia.ceph.com 2023-10-31T12:30:31.188 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:30:31.189 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/write/json_output.0.smithi061.front.sepia.ceph.com of=/dev/stdout 2023-10-31T12:30:31.212 INFO:tasks.cbt_performance:reading total cpu cycles 2023-10-31T12:30:31.213 DEBUG:teuthology.orchestra.run.smithi061:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/write -name 'perf_stat.*' 2023-10-31T12:30:31.267 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-10-31T12:30:31.268 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-10-31T12:30:31.268 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:30:31.268 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/write/json_output.1.smithi061.front.sepia.ceph.com of=/dev/stdout 2023-10-31T12:30:31.323 INFO:tasks.cbt_performance:reading total cpu cycles 2023-10-31T12:30:31.323 DEBUG:teuthology.orchestra.run.smithi061:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/write -name 'perf_stat.*' 2023-10-31T12:30:31.379 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-10-31T12:30:31.379 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-10-31T12:30:31.380 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:30:31.380 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/seq/json_output.0.smithi061.front.sepia.ceph.com of=/dev/stdout 2023-10-31T12:30:31.434 INFO:tasks.cbt_performance:reading total cpu cycles 2023-10-31T12:30:31.435 DEBUG:teuthology.orchestra.run.smithi061:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/seq -name 'perf_stat.*' 2023-10-31T12:30:31.489 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-10-31T12:30:31.490 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-10-31T12:30:31.490 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2023-10-31T12:30:31.490 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/seq/json_output.1.smithi061.front.sepia.ceph.com of=/dev/stdout 2023-10-31T12:30:31.544 INFO:tasks.cbt_performance:reading total cpu cycles 2023-10-31T12:30:31.544 DEBUG:teuthology.orchestra.run.smithi061:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-31cd68af/seq -name 'perf_stat.*' 2023-10-31T12:30:31.599 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-10-31T12:30:31.599 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-10-31T12:30:31.608 INFO:tasks.cbt_performance:Data inserted successfully. 2023-10-31T12:30:31.615 INFO:tasks.cbt_performance:Data inserted successfully. 2023-10-31T12:30:31.622 INFO:tasks.cbt_performance:Data inserted successfully. 2023-10-31T12:30:31.629 INFO:tasks.cbt_performance:Data inserted successfully. 2023-10-31T12:30:31.629 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-31T12:30:31.642 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-31T12:30:31.756 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:30:31.748+0000 7f0ec719e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:30:31.758 INFO:teuthology.orchestra.run.smithi061.stderr:2023-10-31T12:30:31.756+0000 7f0ec719e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-10-31T12:30:31.961 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-31T12:30:31.962 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-31T12:30:31.962 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-31T12:30:31.963 INFO:teuthology.orchestra.run:waiting for 300 2023-10-31T12:30:32.061 INFO:tasks.ceph.osd.0:Stopped 2023-10-31T12:30:32.062 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-31T12:30:32.062 INFO:teuthology.orchestra.run:waiting for 300 2023-10-31T12:30:34.805 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-31T12:30:38.164 INFO:tasks.ceph.osd.1:Stopped 2023-10-31T12:30:38.165 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-31T12:30:38.165 INFO:teuthology.orchestra.run:waiting for 300 2023-10-31T12:30:40.007 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-31T12:30:40.008 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-31T12:30:44.267 INFO:tasks.ceph.osd.2:Stopped 2023-10-31T12:30:44.268 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-31T12:30:44.268 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-31T12:30:44.269 INFO:teuthology.orchestra.run:waiting for 300 2023-10-31T12:30:44.299 INFO:tasks.ceph.mgr.x:Stopped 2023-10-31T12:30:44.299 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-31T12:30:44.300 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-31T12:30:44.300 INFO:teuthology.orchestra.run:waiting for 300 2023-10-31T12:30:44.353 INFO:tasks.ceph.mon.a:Stopped 2023-10-31T12:30:44.354 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-31T12:30:44.354 DEBUG:teuthology.orchestra.run.smithi061:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-10-31T12:30:44.441 INFO:teuthology.orchestra.run.smithi061.stdout:2023-10-31T12:29:46.967271+0000 mon.a (mon.0) 152 : cluster [WRN] Health check failed: Degraded data redundancy: 104/9992 objects degraded (1.041%), 5 pgs degraded (PG_DEGRADED) 2023-10-31T12:30:44.442 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-10-31T12:30:44.442 DEBUG:teuthology.orchestra.run.smithi061:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-10-31T12:30:44.478 DEBUG:teuthology.orchestra.run.smithi061:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-10-31T12:30:44.553 DEBUG:teuthology.orchestra.run.smithi061:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-10-31T12:30:44.627 INFO:teuthology.orchestra.run.smithi061.stdout:2023-10-31T12:29:46.967271+0000 mon.a (mon.0) 152 : cluster [WRN] Health check failed: Degraded data redundancy: 104/9992 objects degraded (1.041%), 5 pgs degraded (PG_DEGRADED) 2023-10-31T12:30:44.627 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:30:44.628 DEBUG:teuthology.orchestra.run.smithi061:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-31T12:30:45.009 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:30:45.009 DEBUG:teuthology.orchestra.run.smithi061:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-31T12:30:45.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-10-31T12:30:45.092 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:30:45.093 DEBUG:teuthology.orchestra.run.smithi061:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-31T12:30:45.149 INFO:tasks.ceph:Archiving mon data... 2023-10-31T12:30:45.150 DEBUG:teuthology.misc:Transferring archived files from smithi061:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7441974/data/mon.a.tgz 2023-10-31T12:30:45.151 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2023-10-31T12:30:45.182 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.dlI1ueoOHG 2023-10-31T12:30:45.183 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.dlI1ueoOHG 2023-10-31T12:30:45.333 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.dlI1ueoOHG 2023-10-31T12:30:45.441 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.dlI1ueoOHG is 179KB 2023-10-31T12:30:45.493 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.dlI1ueoOHG 2023-10-31T12:30:45.508 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-31T12:30:45.508 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-31T12:30:45.577 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-31T12:30:45.578 DEBUG:teuthology.orchestra.run.smithi061:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-31T12:30:45.678 INFO:teuthology.orchestra.run.smithi061.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-31T12:30:45.682 INFO:tasks.ceph:Archiving crash dumps... 2023-10-31T12:30:45.684 DEBUG:teuthology.misc:Transferring archived files from smithi061:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7441974/remote/smithi061/crash 2023-10-31T12:30:45.685 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-31T12:30:45.714 INFO:tasks.ceph:Compressing logs... 2023-10-31T12:30:45.715 DEBUG:teuthology.orchestra.run.smithi061:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-31T12:30:45.792 INFO:teuthology.orchestra.run.smithi061.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-31T12:30:45.792 INFO:teuthology.orchestra.run.smithi061.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.79752.log 2023-10-31T12:30:45.793 INFO:teuthology.orchestra.run.smithi061.stderr:gzip/var/log/ceph/ceph-mon.a.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.83453.log 2023-10-31T12:30:45.793 INFO:teuthology.orchestra.run.smithi061.stderr:gzip -5 --verbose --/var/log/ceph/ceph.tmp-client.admin.79752.log: /var/log/ceph/ceph-client.admin.83452.log 2023-10-31T12:30:45.793 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.83453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81349.log 2023-10-31T12:30:45.793 INFO:teuthology.orchestra.run.smithi061.stderr:gzip -5/var/log/ceph/ceph-client.admin.83452.log: --verbose -- /var/log/ceph/ceph-client.admin.83608.log 2023-10-31T12:30:45.802 INFO:teuthology.orchestra.run.smithi061.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81984.log 2023-10-31T12:30:45.802 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.81349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82563.log 2023-10-31T12:30:45.802 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.83608.log: 66.1% -- replaced with /var/log/ceph/ceph-client.admin.83608.log.gz 2023-10-31T12:30:45.802 INFO:teuthology.orchestra.run.smithi061.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82185.log 2023-10-31T12:30:45.803 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.81984.log: 66.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.79752.log.gz 2023-10-31T12:30:45.803 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.82563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83342.log 2023-10-31T12:30:45.803 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.82185.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-31T12:30:45.803 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.83342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82307.log 2023-10-31T12:30:45.803 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82227.log 2023-10-31T12:30:45.804 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.82307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82435.log 2023-10-31T12:30:45.804 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.82227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81199.log 2023-10-31T12:30:45.804 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.82435.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-31T12:30:45.804 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.81199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82082.log 2023-10-31T12:30:45.804 INFO:teuthology.orchestra.run.smithi061.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83388.log 2023-10-31T12:30:45.805 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.82082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82521.log 2023-10-31T12:30:45.805 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.83388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83609.log 2023-10-31T12:30:45.805 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.82521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82434.log 2023-10-31T12:30:45.805 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.83609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82143.log 2023-10-31T12:30:45.805 INFO:teuthology.orchestra.run.smithi061.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.83609.log.gz 2023-10-31T12:30:45.806 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.82434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83313.log 2023-10-31T12:30:45.806 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.82143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82040.log 2023-10-31T12:30:45.806 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.83313.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-31T12:30:45.806 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.82040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81309.log 2023-10-31T12:30:45.806 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82605.log 2023-10-31T12:30:45.807 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.81309.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-31T12:30:45.807 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.82605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82308.log 2023-10-31T12:30:45.807 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81546.log 2023-10-31T12:30:45.807 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.82308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82309.log 2023-10-31T12:30:45.807 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.81546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83909.log 2023-10-31T12:30:45.808 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.82309.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-31T12:30:45.808 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.83909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81575.log 2023-10-31T12:30:45.808 INFO:teuthology.orchestra.run.smithi061.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83909.log.gz 2023-10-31T12:30:45.808 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-31T12:30:45.808 INFO:teuthology.orchestra.run.smithi061.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.83453.log.gz 2023-10-31T12:30:45.809 INFO:teuthology.orchestra.run.smithi061.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.83452.log.gz 2023-10-31T12:30:45.809 INFO:teuthology.orchestra.run.smithi061.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83251.log 2023-10-31T12:30:45.809 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.81575.log: 85.0% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-31T12:30:45.809 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82395.log 2023-10-31T12:30:45.809 INFO:teuthology.orchestra.run.smithi061.stderr:gzip/var/log/ceph/ceph-client.admin.83251.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.81389.log 2023-10-31T12:30:45.810 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.82395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81111.log 2023-10-31T12:30:45.810 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.81389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83365.log 2023-10-31T12:30:45.810 INFO:teuthology.orchestra.run.smithi061.stderr: 87.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-31T12:30:45.811 INFO:teuthology.orchestra.run.smithi061.stderr:/var/log/ceph/ceph-client.admin.81111.log: /var/log/ceph/ceph-client.admin.83365.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.81199.log.gz 2023-10-31T12:30:45.811 INFO:teuthology.orchestra.run.smithi061.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83388.log.gz 2023-10-31T12:30:45.813 INFO:teuthology.orchestra.run.smithi061.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81309.log.gz 2023-10-31T12:30:45.814 INFO:teuthology.orchestra.run.smithi061.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81349.log.gz 2023-10-31T12:30:45.814 INFO:teuthology.orchestra.run.smithi061.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81389.log.gz 2023-10-31T12:30:45.816 INFO:teuthology.orchestra.run.smithi061.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81546.log.gz 2023-10-31T12:30:45.818 INFO:teuthology.orchestra.run.smithi061.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81575.log.gz 2023-10-31T12:30:45.818 INFO:teuthology.orchestra.run.smithi061.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83365.log.gz 2023-10-31T12:30:45.822 INFO:teuthology.orchestra.run.smithi061.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81111.log.gz 2023-10-31T12:30:45.822 INFO:teuthology.orchestra.run.smithi061.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-31T12:30:45.832 INFO:teuthology.orchestra.run.smithi061.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81984.log.gz 2023-10-31T12:30:45.833 INFO:teuthology.orchestra.run.smithi061.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82040.log.gz 2023-10-31T12:30:45.833 INFO:teuthology.orchestra.run.smithi061.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82082.log.gz 2023-10-31T12:30:45.833 INFO:teuthology.orchestra.run.smithi061.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82143.log.gz 2023-10-31T12:30:45.833 INFO:teuthology.orchestra.run.smithi061.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82185.log.gz 2023-10-31T12:30:45.834 INFO:teuthology.orchestra.run.smithi061.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82227.log.gz 2023-10-31T12:30:45.834 INFO:teuthology.orchestra.run.smithi061.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82308.log.gz 2023-10-31T12:30:45.834 INFO:teuthology.orchestra.run.smithi061.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82307.log.gz 2023-10-31T12:30:45.834 INFO:teuthology.orchestra.run.smithi061.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82309.log.gz 2023-10-31T12:30:45.834 INFO:teuthology.orchestra.run.smithi061.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82395.log.gz 2023-10-31T12:30:45.835 INFO:teuthology.orchestra.run.smithi061.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82434.log.gz 2023-10-31T12:30:45.835 INFO:teuthology.orchestra.run.smithi061.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82435.log.gz 2023-10-31T12:30:45.835 INFO:teuthology.orchestra.run.smithi061.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82521.log.gz 2023-10-31T12:30:45.835 INFO:teuthology.orchestra.run.smithi061.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82563.log.gz 2023-10-31T12:30:45.835 INFO:teuthology.orchestra.run.smithi061.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82605.log.gz 2023-10-31T12:30:45.858 INFO:teuthology.orchestra.run.smithi061.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83251.log.gz 2023-10-31T12:30:45.858 INFO:teuthology.orchestra.run.smithi061.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.83313.log.gz 2023-10-31T12:30:45.859 INFO:teuthology.orchestra.run.smithi061.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83342.log.gz 2023-10-31T12:30:46.036 INFO:teuthology.orchestra.run.smithi061.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-31T12:30:51.273 INFO:teuthology.orchestra.run.smithi061.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-31T12:30:51.987 INFO:teuthology.orchestra.run.smithi061.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-31T12:30:52.015 INFO:teuthology.orchestra.run.smithi061.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-31T12:30:52.017 INFO:teuthology.orchestra.run.smithi061.stderr: 2023-10-31T12:30:52.017 INFO:teuthology.orchestra.run.smithi061.stderr:real 0m6.248s 2023-10-31T12:30:52.018 INFO:teuthology.orchestra.run.smithi061.stderr:user 0m3.197s 2023-10-31T12:30:52.018 INFO:teuthology.orchestra.run.smithi061.stderr:sys 0m0.205s 2023-10-31T12:30:52.018 INFO:tasks.ceph:Archiving logs... 2023-10-31T12:30:52.018 DEBUG:teuthology.misc:Transferring archived files from smithi061:/var/log/ceph to /home/teuthworker/archive/matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7441974/remote/smithi061/log 2023-10-31T12:30:52.020 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-31T12:30:52.584 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-31T12:30:52.594 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-31T12:30:52.594 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-31T12:30:52.651 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-31T12:30:52.663 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-31T12:30:52.663 DEBUG:teuthology.orchestra.run.smithi061:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-31T12:30:52.681 INFO:teuthology.orchestra.run.smithi061.stderr:bash: line 1: ntpq: command not found 2023-10-31T12:30:52.757 INFO:teuthology.orchestra.run.smithi061.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-31T12:30:52.757 INFO:teuthology.orchestra.run.smithi061.stdout:=============================================================================== 2023-10-31T12:30:52.758 INFO:teuthology.orchestra.run.smithi061.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 25 +835us[ +843us] +/- 80ms 2023-10-31T12:30:52.758 INFO:teuthology.orchestra.run.smithi061.stdout:^* hv02.front.sepia.ceph.com 3 6 377 25 +537us[ +544us] +/- 65ms 2023-10-31T12:30:52.758 INFO:teuthology.orchestra.run.smithi061.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 25 -1368us[-1368us] +/- 75ms 2023-10-31T12:30:52.758 INFO:teuthology.orchestra.run.smithi061.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-10-31T12:30:52.777 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-31T12:30:52.786 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-31T12:30:52.787 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-31T12:30:52.796 DEBUG:teuthology.orchestra.run.smithi061:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-31T12:30:52.981 DEBUG:teuthology.orchestra.run.smithi061:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-31T12:30:53.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-31T12:30:53.017 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-31T12:30:53.026 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-31T12:30:53.034 INFO:teuthology.task.internal:Duration was 755.177352 seconds 2023-10-31T12:30:53.035 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-31T12:30:53.044 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-31T12:30:53.044 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-31T12:30:53.216 INFO:teuthology.orchestra.run.smithi061.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-31T12:30:53.622 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-31T12:30:53.623 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi061.front.sepia.ceph.com 2023-10-31T12:30:53.623 DEBUG:teuthology.orchestra.run.smithi061:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-31T12:30:53.645 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-31T12:30:53.645 DEBUG:teuthology.orchestra.run.smithi061:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-31T12:30:53.719 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-31T12:30:53.719 DEBUG:teuthology.orchestra.run.smithi061:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-31T12:30:53.863 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-31T12:30:53.873 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-31T12:30:53.874 DEBUG:teuthology.orchestra.run.smithi061:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-31T12:30:53.916 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-31T12:30:53.929 DEBUG:teuthology.orchestra.run.smithi061:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-31T12:30:54.000 INFO:teuthology.orchestra.run.smithi061.stdout:kernel.core_pattern = core 2023-10-31T12:30:54.023 DEBUG:teuthology.orchestra.run.smithi061:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-31T12:30:54.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-31T12:30:54.079 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-31T12:30:54.089 INFO:teuthology.task.internal:Transferring archived files... 2023-10-31T12:30:54.090 DEBUG:teuthology.misc:Transferring archived files from smithi061:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7441974/remote/smithi061 2023-10-31T12:30:54.090 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-31T12:30:54.319 INFO:teuthology.task.internal:Removing archive directory... 2023-10-31T12:30:54.319 DEBUG:teuthology.orchestra.run.smithi061:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-31T12:30:54.372 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-31T12:30:54.391 INFO:teuthology.task.internal:Not uploading archives. 2023-10-31T12:30:54.391 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-31T12:30:54.401 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-31T12:30:54.401 DEBUG:teuthology.orchestra.run.smithi061:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-31T12:30:54.428 INFO:teuthology.orchestra.run.smithi061.stdout: 265673 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 31 12:30 /home/ubuntu/cephtest 2023-10-31T12:30:54.429 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-31T12:30:54.440 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-31T12:30:54.461 INFO:teuthology.nuke:Checking targets against current locks 2023-10-31T12:30:54.487 DEBUG:teuthology.nuke:shortname: smithi061 2023-10-31T12:30:54.488 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-31T12:30:54.511 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi061.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-10-31_11:40:33-crimson-rados-wip-wip-matanb-crimson-do_osd_ops_execute-v3-distro-crimson-smithi/7441974', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-31 12:10:00.948266', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLrJA40Hkx/RPJJwnSiZcODOwfqW/iC7Q/PzuwzQah+m/ONKi/hvZW56fzMf8XhGcoKh5u3N8Bc1cNqBLauDHrk='} 2023-10-31T12:30:54.563 INFO:teuthology.orchestra.console.smithi061:Power off 2023-10-31T12:30:54.564 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-31T12:30:54.587 DEBUG:teuthology.orchestra.console.smithi061:power off output: Chassis Power Control: Down/Off 2023-10-31T12:30:54.588 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-31T12:30:54.614 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is on 2023-10-31T12:30:58.616 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-31T12:30:58.725 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is on 2023-10-31T12:31:02.729 DEBUG:teuthology.orchestra.console.smithi061:pexpect command: ipmitool -H smithi061.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-31T12:31:02.839 DEBUG:teuthology.orchestra.console.smithi061:check power output: Chassis Power is off 2023-10-31T12:31:02.940 INFO:teuthology.orchestra.console.smithi061:Power off completed 2023-10-31T12:31:03.045 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/Rfy7LkB4k/rados-bench?orgId=1&var-branch_name=wip-wip-matanb-crimson-do_osd_ops_execute-v3&var-job_id_selected=7441974 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_seq_read} duration: 755.1773521900177 failure_reason: '"2023-10-31T12:29:46.967271+0000 mon.a (mon.0) 152 : cluster [WRN] Health check failed: Degraded data redundancy: 104/9992 objects degraded (1.041%), 5 pgs degraded (PG_DEGRADED)" in cluster log' flavor: crimson owner: scheduled_matan@teuthology success: false 2023-10-31T12:31:03.045 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-31T12:31:03.115 INFO:teuthology.run:FAIL