2023-09-07T10:51:36.760 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-07T10:51:36.761 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name matan-2023-09-07_10:45:43-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-ref-crimson-only-distro-crimson-smithi --archive /home/teuthworker/archive/matan-2023-09-07_10:45:43-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-ref-crimson-only-distro-crimson-smithi/7390804 --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} --owner scheduled_matan@teuthology -- /home/teuthworker/archive/matan-2023-09-07_10:45:43-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-ref-crimson-only-distro-crimson-smithi/7390804/orig.config.yaml 2023-09-07T10:51:36.802 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-07T10:51:36.884 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2023-09-07_10:45:43-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-ref-crimson-only-distro-crimson-smithi/7390804 branch: wip-matanb-crimson-do_osd_ops_execute-ref-crimson-only description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} email: null first_in_suite: false job_id: '7390804' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2023-09-07_10:45:43-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-ref-crimson-only-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-do_osd_ops_execute-ref-crimson-only ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: 7075f69375d3b106d3a94dda948a86d616371099 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: 7075f69375d3b106d3a94dda948a86d616371099 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: f4c5809eee86803e43be24d3362e60639f88c948 owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1657 sha1: 7075f69375d3b106d3a94dda948a86d616371099 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_f4c5809eee86803e43be24d3362e60639f88c948/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: f4c5809eee86803e43be24d3362e60639f88c948 targets: smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGaX5e7rZpui2YElhU102jZW5H2XNqBZ4e0SP9T451dXzh+rj/ICxhb9YL6Yl+7gbPr8wbmGmold07FAy9yrfRk= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4194304 pool_monitoring_list: - collectl pool_profile: replicated readmode: rand run_monitoring_list: - collectl time: 60 write_only: false cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-07_10:45:43 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1394 2023-09-07T10:51:36.884 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_f4c5809eee86803e43be24d3362e60639f88c948/qa; will attempt to use it 2023-09-07T10:51:36.885 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_f4c5809eee86803e43be24d3362e60639f88c948/qa/tasks 2023-09-07T10:51:36.885 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-07T10:51:36.887 INFO:teuthology.task.internal:Checking packages... 2023-09-07T10:51:36.911 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '7075f69375d3b106d3a94dda948a86d616371099' 2023-09-07T10:51:36.911 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-07T10:51:36.911 INFO:teuthology.packaging:ref: None 2023-09-07T10:51:36.912 INFO:teuthology.packaging:tag: None 2023-09-07T10:51:36.912 INFO:teuthology.packaging:branch: wip-matanb-crimson-do_osd_ops_execute-ref-crimson-only 2023-09-07T10:51:36.912 INFO:teuthology.packaging:sha1: 7075f69375d3b106d3a94dda948a86d616371099 2023-09-07T10:51:36.913 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-do_osd_ops_execute-ref-crimson-only 2023-09-07T10:51:37.129 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-5797.g7075f693 2023-09-07T10:51:37.129 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-07T10:51:37.136 INFO:teuthology.task.internal:no buildpackages task found 2023-09-07T10:51:37.136 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-07T10:51:37.151 INFO:teuthology.task.internal:Saving configuration 2023-09-07T10:51:37.166 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-07T10:51:37.173 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-07T10:51:37.198 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-09-07_10:45:43-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-ref-crimson-only-distro-crimson-smithi/7390804', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-07 10:46:58.224355', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGaX5e7rZpui2YElhU102jZW5H2XNqBZ4e0SP9T451dXzh+rj/ICxhb9YL6Yl+7gbPr8wbmGmold07FAy9yrfRk='} 2023-09-07T10:51:37.198 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-07T10:51:37.205 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-09-07T10:51:37.205 INFO:teuthology.run_tasks:Running task console_log... 2023-09-07T10:51:37.243 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdca91ab940>, signals=[15]) 2023-09-07T10:51:37.244 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-07T10:51:37.250 INFO:teuthology.task.internal:Opening connections... 2023-09-07T10:51:37.250 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2023-09-07T10:51:37.252 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-07T10:51:37.352 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-07T10:51:37.359 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2023-09-07T10:51:37.423 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2023-09-07T10:51:37.423 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2023-09-07T10:51:37.478 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="CentOS Stream" 2023-09-07T10:51:37.478 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="9" 2023-09-07T10:51:37.478 INFO:teuthology.orchestra.run.smithi031.stdout:ID="centos" 2023-09-07T10:51:37.478 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="rhel fedora" 2023-09-07T10:51:37.479 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="9" 2023-09-07T10:51:37.479 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el9" 2023-09-07T10:51:37.479 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="CentOS Stream 9" 2023-09-07T10:51:37.479 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2023-09-07T10:51:37.479 INFO:teuthology.orchestra.run.smithi031.stdout:LOGO="fedora-logo-icon" 2023-09-07T10:51:37.480 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-09-07T10:51:37.480 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://centos.org/" 2023-09-07T10:51:37.480 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-07T10:51:37.480 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-09-07T10:51:37.480 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-09-07T10:51:37.481 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2023-09-07T10:51:37.509 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-07T10:51:37.518 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-07T10:51:37.524 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-07T10:51:37.524 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2023-09-07T10:51:37.541 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-07T10:51:37.548 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-07T10:51:37.548 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2023-09-07T10:51:37.607 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-07T10:51:37.638 INFO:teuthology.run_tasks:Running task kernel... 2023-09-07T10:51:37.653 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-07T10:51:37.654 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-07T10:51:37.654 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-07T10:51:37.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-07T10:51:37.672 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2023-09-07T10:51:37.727 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-361.el9.x86_64 2023-09-07T10:51:37.727 INFO:teuthology.task.kernel:Running kernel on smithi031: 5.14.0-361.el9.x86_64 2023-09-07T10:51:37.728 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum install -y kernel 2023-09-07T10:51:39.745 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - AppStream 13 MB/s | 6.3 MB 00:00 2023-09-07T10:51:41.087 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - BaseOS 7.6 MB/s | 2.0 MB 00:00 2023-09-07T10:51:42.977 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 7.8 MB 00:01 2023-09-07T10:51:44.936 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 20 MB/s | 18 MB 00:00 2023-09-07T10:51:50.495 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 5.7 MB 00:02 2023-09-07T10:51:51.837 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 14 kB 00:00 2023-09-07T10:51:53.572 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 11 MB/s | 19 MB 00:01 2023-09-07T10:51:58.105 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2023-09-07T10:52:00.103 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-09-07T10:52:00.104 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-09-07T10:52:00.193 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T10:52:00.203 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T10:52:00.203 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-09-07T10:52:00.203 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T10:52:00.204 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-07T10:52:00.204 INFO:teuthology.orchestra.run.smithi031.stdout: kernel x86_64 5.14.0-363.el9 CentOS-BaseOS 5.0 M 2023-09-07T10:52:00.204 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-09-07T10:52:00.204 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-core x86_64 5.14.0-363.el9 CentOS-BaseOS 20 M 2023-09-07T10:52:00.204 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules x86_64 5.14.0-363.el9 CentOS-BaseOS 37 M 2023-09-07T10:52:00.204 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-core x86_64 5.14.0-363.el9 CentOS-BaseOS 32 M 2023-09-07T10:52:00.205 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:52:00.205 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-07T10:52:00.205 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T10:52:00.205 INFO:teuthology.orchestra.run.smithi031.stdout:Install 4 Packages 2023-09-07T10:52:00.205 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:52:00.206 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 93 M 2023-09-07T10:52:00.206 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 120 M 2023-09-07T10:52:00.206 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-07T10:52:00.531 INFO:teuthology.orchestra.run.smithi031.stdout:(1/4): kernel-5.14.0-363.el9.x86_64.rpm 15 MB/s | 5.0 MB 00:00 2023-09-07T10:52:00.998 INFO:teuthology.orchestra.run.smithi031.stdout:(2/4): kernel-core-5.14.0-363.el9.x86_64.rpm 25 MB/s | 20 MB 00:00 2023-09-07T10:52:01.590 INFO:teuthology.orchestra.run.smithi031.stdout:(3/4): kernel-modules-core-5.14.0-363.el9.x86_6 30 MB/s | 32 MB 00:01 2023-09-07T10:52:02.015 INFO:teuthology.orchestra.run.smithi031.stdout:(4/4): kernel-modules-5.14.0-363.el9.x86_64.rpm 20 MB/s | 37 MB 00:01 2023-09-07T10:52:02.018 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-07T10:52:02.019 INFO:teuthology.orchestra.run.smithi031.stdout:Total 51 MB/s | 93 MB 00:01 2023-09-07T10:52:02.019 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-07T10:52:02.453 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-07T10:52:02.453 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-07T10:52:03.447 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-07T10:52:03.447 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-07T10:52:05.934 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-07T10:52:06.592 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-modules-core-5.14.0-363.el9.x86_64 1/4 2023-09-07T10:52:06.715 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-core-5.14.0-363.el9.x86_64 2/4 2023-09-07T10:52:09.614 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-core-5.14.0-363.el9.x86_64 2/4 2023-09-07T10:52:09.791 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-modules-5.14.0-363.el9.x86_64 3/4 2023-09-07T10:52:14.736 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-5.14.0-363.el9.x86_64 3/4 2023-09-07T10:52:14.825 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-5.14.0-363.el9.x86_64 4/4 2023-09-07T10:52:19.801 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-core-5.14.0-363.el9.x86_64 4/4 2023-09-07T10:52:50.815 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-core-5.14.0-363.el9.x86_64 4/4 2023-09-07T10:52:50.860 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-5.14.0-363.el9.x86_64 4/4 2023-09-07T10:52:51.274 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-5.14.0-363.el9.x86_64 4/4 2023-09-07T10:52:51.275 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-5.14.0-363.el9.x86_64 1/4 2023-09-07T10:52:51.275 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-core-5.14.0-363.el9.x86_64 2/4 2023-09-07T10:52:51.275 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-modules-5.14.0-363.el9.x86_64 3/4 2023-09-07T10:52:51.751 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-modules-core-5.14.0-363.el9.x86_64 4/4 2023-09-07T10:52:51.752 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:52:51.752 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-07T10:52:51.752 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-5.14.0-363.el9.x86_64 2023-09-07T10:52:51.752 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-core-5.14.0-363.el9.x86_64 2023-09-07T10:52:51.753 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-5.14.0-363.el9.x86_64 2023-09-07T10:52:51.753 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-core-5.14.0-363.el9.x86_64 2023-09-07T10:52:51.753 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:52:51.753 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T10:52:52.246 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2023-09-07T10:52:52.372 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-363.el9.x86_64 2023-09-07T10:52:52.373 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-363.el9.x86_64 2023-09-07T10:52:52.374 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-363.el9.x86_64 2023-09-07T10:52:52.374 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-07T10:52:52.374 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-07T10:52:52.374 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-09-07T10:52:52.375 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi031.front.sepia.ceph.com, path=None, version=distro) 2023-09-07T10:52:52.375 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2023-09-07T10:52:52.480 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-363.el9.x86_64 2023-09-07T10:52:52.481 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-363.el9.x86_64 2023-09-07T10:52:52.482 DEBUG:teuthology.orchestra.run.smithi031:> sudo rpm -qi grub2-tools 2023-09-07T10:52:52.525 INFO:teuthology.orchestra.run.smithi031.stdout:Name : grub2-tools 2023-09-07T10:52:52.525 INFO:teuthology.orchestra.run.smithi031.stdout:Epoch : 1 2023-09-07T10:52:52.525 INFO:teuthology.orchestra.run.smithi031.stdout:Version : 2.06 2023-09-07T10:52:52.525 INFO:teuthology.orchestra.run.smithi031.stdout:Release : 68.el9 2023-09-07T10:52:52.525 INFO:teuthology.orchestra.run.smithi031.stdout:Architecture: x86_64 2023-09-07T10:52:52.526 INFO:teuthology.orchestra.run.smithi031.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-09-07T10:52:52.526 INFO:teuthology.orchestra.run.smithi031.stdout:Group : Unspecified 2023-09-07T10:52:52.526 INFO:teuthology.orchestra.run.smithi031.stdout:Size : 8264254 2023-09-07T10:52:52.526 INFO:teuthology.orchestra.run.smithi031.stdout:License : GPLv3+ 2023-09-07T10:52:52.527 INFO:teuthology.orchestra.run.smithi031.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-09-07T10:52:52.527 INFO:teuthology.orchestra.run.smithi031.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-09-07T10:52:52.527 INFO:teuthology.orchestra.run.smithi031.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-09-07T10:52:52.527 INFO:teuthology.orchestra.run.smithi031.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-09-07T10:52:52.527 INFO:teuthology.orchestra.run.smithi031.stdout:Packager : builder@centos.org 2023-09-07T10:52:52.528 INFO:teuthology.orchestra.run.smithi031.stdout:Vendor : CentOS 2023-09-07T10:52:52.528 INFO:teuthology.orchestra.run.smithi031.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-07T10:52:52.528 INFO:teuthology.orchestra.run.smithi031.stdout:Summary : Support tools for GRUB. 2023-09-07T10:52:52.528 INFO:teuthology.orchestra.run.smithi031.stdout:Description : 2023-09-07T10:52:52.528 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:52:52.529 INFO:teuthology.orchestra.run.smithi031.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-07T10:52:52.529 INFO:teuthology.orchestra.run.smithi031.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-07T10:52:52.529 INFO:teuthology.orchestra.run.smithi031.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-07T10:52:52.529 INFO:teuthology.orchestra.run.smithi031.stdout:hardware devices. 2023-09-07T10:52:52.529 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:52:52.529 INFO:teuthology.orchestra.run.smithi031.stdout:This subpackage provides tools for support of all platforms. 2023-09-07T10:52:52.530 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-07T10:52:52.531 INFO:teuthology.task.kernel:Updating grub on smithi031 to boot 5.14.0-363.el9.x86_64 2023-09-07T10:52:52.531 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-07T10:52:53.277 INFO:teuthology.orchestra.run.smithi031.stderr:Generating grub configuration file ... 2023-09-07T10:52:54.729 INFO:teuthology.orchestra.run.smithi031.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-09-07T10:52:54.756 INFO:teuthology.orchestra.run.smithi031.stderr:done 2023-09-07T10:52:54.758 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2023-09-07T10:52:54.773 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.dOnMBsPMNV 2023-09-07T10:52:54.773 DEBUG:teuthology.orchestra.run.smithi031:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.dOnMBsPMNV 2023-09-07T10:52:54.845 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.dOnMBsPMNV 2023-09-07T10:52:55.101 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.dOnMBsPMNV is 6KB 2023-09-07T10:52:55.150 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.dOnMBsPMNV 2023-09-07T10:52:55.165 DEBUG:teuthology.orchestra.run.smithi031:> sudo /bin/ls /boot/loader/entries || true 2023-09-07T10:52:55.242 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-09-07T10:52:55.242 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-09-07T10:52:55.243 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-09-07T10:52:55.243 INFO:teuthology.orchestra.run.smithi031.stdout:15949e3c88704300882482eddfd7baae-5.14.0-363.el9.x86_64.conf 2023-09-07T10:52:55.244 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-363.el9.x86_64 2023-09-07T10:52:55.409 DEBUG:teuthology.orchestra.run.smithi031:> sudo shutdown -r now 2023-09-07T10:53:25.444 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-09-07T10:53:25.445 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-09-07T10:53:25.446 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-07T10:54:25.490 DEBUG:teuthology.orchestra.remote:timed out 2023-09-07T10:54:34.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-09-07T10:54:34.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-07T10:54:34.797 DEBUG:teuthology.orchestra.run.smithi031:> true 2023-09-07T10:54:35.153 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-09-07T10:54:35.153 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-363.el9.x86_64"... 2023-09-07T10:54:35.153 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2023-09-07T10:54:35.168 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-363.el9.x86_64 2023-09-07T10:54:35.168 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-363.el9.x86_64 vs 5.14.0-363.el9.x86_64 2023-09-07T10:54:35.168 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-07T10:54:35.169 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-09-07T10:54:36.169 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-09-07T10:54:36.170 DEBUG:teuthology.orchestra.run.smithi031:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-07T10:54:36.235 INFO:teuthology.orchestra.run.smithi031.stdout:ttyS1 2023-09-07T10:54:36.261 DEBUG:teuthology.parallel:result is None 2023-09-07T10:54:36.262 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-07T10:54:36.268 INFO:teuthology.task.internal:Creating test directory... 2023-09-07T10:54:36.269 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-07T10:54:36.286 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-07T10:54:36.292 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-07T10:54:36.301 INFO:teuthology.task.internal:Creating archive directory... 2023-09-07T10:54:36.301 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-07T10:54:36.357 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-07T10:54:36.365 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-07T10:54:36.366 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-07T10:54:36.430 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-07T10:54:36.447 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-07T10:54:36.449 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-07T10:54:36.456 INFO:teuthology.task.internal:Configuring sudo... 2023-09-07T10:54:36.456 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-07T10:54:36.522 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-07T10:54:36.530 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-07T10:54:36.531 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-07T10:54:36.578 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-07T10:54:36.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-07T10:54:36.727 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-07T10:54:36.771 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-07T10:54:36.818 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-07T10:54:36.819 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-07T10:54:36.891 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2023-09-07T10:54:36.982 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-07T10:54:37.237 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-07T10:54:37.245 INFO:teuthology.task.internal:Starting timer... 2023-09-07T10:54:37.245 INFO:teuthology.run_tasks:Running task pcp... 2023-09-07T10:54:37.255 INFO:teuthology.run_tasks:Running task selinux... 2023-09-07T10:54:37.271 DEBUG:teuthology.orchestra.run.smithi031:> sudo service auditd rotate 2023-09-07T10:54:37.332 INFO:teuthology.orchestra.run.smithi031.stdout:Rotating logs: 2023-09-07T10:54:37.334 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-07T10:54:37.334 DEBUG:teuthology.orchestra.run.smithi031:> /usr/sbin/getenforce 2023-09-07T10:54:37.363 INFO:teuthology.orchestra.run.smithi031.stdout:Permissive 2023-09-07T10:54:37.364 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi031.front.sepia.ceph.com': 'permissive'} 2023-09-07T10:54:37.364 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-07T10:54:37.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-07T10:54:37.438 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-07T10:54:37.438 DEBUG:teuthology.orchestra.run.smithi031:> sudo /usr/sbin/setenforce permissive 2023-09-07T10:54:37.532 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-07T10:54:37.542 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-09-07T10:54:37.542 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-07T10:54:37.558 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-07T10:54:37.559 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi031.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-07T10:57:04.901 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi031.front.sepia.ceph.com')] 2023-09-07T10:57:04.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-09-07T10:57:04.904 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-07T10:57:05.003 DEBUG:teuthology.orchestra.run.smithi031:> true 2023-09-07T10:57:05.069 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-09-07T10:57:05.069 INFO:teuthology.run_tasks:Running task clock... 2023-09-07T10:57:05.082 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-07T10:57:05.082 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-07T10:57:05.083 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-07T10:57:05.144 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-07T10:57:05.164 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-07T10:57:05.200 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: ntpd: command not found 2023-09-07T10:57:05.217 INFO:teuthology.orchestra.run.smithi031.stdout:506 Cannot talk to daemon 2023-09-07T10:57:05.236 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-07T10:57:05.256 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-07T10:57:05.303 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2023-09-07T10:57:05.306 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-07T10:57:05.307 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2023-09-07T10:57:05.307 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-07T10:57:05.307 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-07T10:57:05.307 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-07T10:57:05.307 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-07T10:57:05.308 INFO:teuthology.run_tasks:Running task install... 2023-09-07T10:57:05.316 DEBUG:teuthology.task.install:project ceph 2023-09-07T10:57:05.317 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '7075f69375d3b106d3a94dda948a86d616371099'}} 2023-09-07T10:57:05.317 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '7075f69375d3b106d3a94dda948a86d616371099'} 2023-09-07T10:57:05.317 INFO:teuthology.task.install:Using flavor: crimson 2023-09-07T10:57:05.326 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-07T10:57:05.327 INFO:teuthology.task.install:extra packages: [] 2023-09-07T10:57:05.329 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7075f69375d3b106d3a94dda948a86d616371099', 'tag': None, 'wait_for_package': False} 2023-09-07T10:57:05.330 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=7075f69375d3b106d3a94dda948a86d616371099 2023-09-07T10:57:05.551 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-do_osd_ops_execute-ref-crimson-only/7075f69375d3b106d3a94dda948a86d616371099/centos/9/flavors/crimson/ 2023-09-07T10:57:05.551 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5797.g7075f693 2023-09-07T10:57:05.731 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-do_osd_ops_execute-ref-crimson-only/7075f69375d3b106d3a94dda948a86d616371099/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-do_osd_ops_execute-ref-crimson-only/7075f69375d3b106d3a94dda948a86d616371099/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-matanb-crimson-do_osd_ops_execute-ref-crimson-only/7075f69375d3b106d3a94dda948a86d616371099/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-07T10:57:05.732 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-07T10:57:05.732 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-07T10:57:05.764 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-07T10:57:05.764 DEBUG:teuthology.orchestra.run.smithi031:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7075f69375d3b106d3a94dda948a86d616371099/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-07T10:57:05.836 DEBUG:teuthology.orchestra.run.smithi031:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-07T10:57:05.923 DEBUG:teuthology.orchestra.run.smithi031:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-07T10:57:05.953 INFO:teuthology.orchestra.run.smithi031.stdout:check_obsoletes = 1 2023-09-07T10:57:05.955 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum clean all 2023-09-07T10:57:06.213 INFO:teuthology.orchestra.run.smithi031.stdout:66 files removed 2023-09-07T10:57:06.240 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-radosgw 2023-09-07T10:57:06.919 INFO:teuthology.orchestra.run.smithi031.stdout:ceph packages for x86_64 234 kB/s | 81 kB 00:00 2023-09-07T10:57:07.269 INFO:teuthology.orchestra.run.smithi031.stdout:ceph noarch packages 31 kB/s | 9.5 kB 00:00 2023-09-07T10:57:07.620 INFO:teuthology.orchestra.run.smithi031.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2023-09-07T10:57:08.005 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - AppStream 18 MB/s | 6.3 MB 00:00 2023-09-07T10:57:09.372 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2023-09-07T10:57:20.380 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 751 kB/s | 7.8 MB 00:10 2023-09-07T10:57:27.246 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 3.0 MB/s | 18 MB 00:05 2023-09-07T10:57:32.283 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 5.7 MB 00:01 2023-09-07T10:57:33.756 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 14 kB 00:00 2023-09-07T10:57:35.076 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 15 MB/s | 19 MB 00:01 2023-09-07T10:57:39.490 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2023-09-07T10:57:41.775 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T10:57:41.778 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T10:57:41.779 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-09-07T10:57:41.779 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T10:57:41.779 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-07T10:57:41.779 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw x86_64 2:18.0.0-5797.g7075f693.el9 ceph 8.2 M 2023-09-07T10:57:41.779 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrading: 2023-09-07T10:57:41.780 INFO:teuthology.orchestra.run.smithi031.stdout: librados2 x86_64 2:18.0.0-5797.g7075f693.el9 ceph 3.1 M 2023-09-07T10:57:41.780 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1 x86_64 2:18.0.0-5797.g7075f693.el9 ceph 3.0 M 2023-09-07T10:57:41.780 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-09-07T10:57:41.780 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options 2023-09-07T10:57:41.780 INFO:teuthology.orchestra.run.smithi031.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-09-07T10:57:41.781 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base x86_64 2:18.0.0-5797.g7075f693.el9 ceph 6.6 M 2023-09-07T10:57:41.781 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common x86_64 2:18.0.0-5797.g7075f693.el9 ceph 17 M 2023-09-07T10:57:41.781 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux x86_64 2:18.0.0-5797.g7075f693.el9 ceph 24 k 2023-09-07T10:57:41.781 INFO:teuthology.orchestra.run.smithi031.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-09-07T10:57:41.781 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-09-07T10:57:41.782 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-09-07T10:57:41.782 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-09-07T10:57:41.782 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2 x86_64 2:18.0.0-5797.g7075f693.el9 ceph 682 k 2023-09-07T10:57:41.782 INFO:teuthology.orchestra.run.smithi031.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-09-07T10:57:41.782 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-09-07T10:57:41.783 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-09-07T10:57:41.783 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1 x86_64 2:18.0.0-5797.g7075f693.el9 ceph 469 k 2023-09-07T10:57:41.783 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-09-07T10:57:41.783 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2 x86_64 2:18.0.0-5797.g7075f693.el9 ceph 4.4 M 2023-09-07T10:57:41.791 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-09-07T10:57:41.791 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-09-07T10:57:41.791 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-09-07T10:57:41.791 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse 2023-09-07T10:57:41.792 INFO:teuthology.orchestra.run.smithi031.stdout: x86_64 2:18.0.0-5797.g7075f693.el9 ceph 45 k 2023-09-07T10:57:41.792 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common x86_64 2:18.0.0-5797.g7075f693.el9 ceph 127 k 2023-09-07T10:57:41.792 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs x86_64 2:18.0.0-5797.g7075f693.el9 ceph 161 k 2023-09-07T10:57:41.792 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-09-07T10:57:41.793 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados x86_64 2:18.0.0-5797.g7075f693.el9 ceph 321 k 2023-09-07T10:57:41.793 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd x86_64 2:18.0.0-5797.g7075f693.el9 ceph 298 k 2023-09-07T10:57:41.793 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw x86_64 2:18.0.0-5797.g7075f693.el9 ceph 99 k 2023-09-07T10:57:41.793 INFO:teuthology.orchestra.run.smithi031.stdout: re2 x86_64 1:20211101-2.el9 epel 192 k 2023-09-07T10:57:41.793 INFO:teuthology.orchestra.run.smithi031.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-09-07T10:57:41.794 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:57:41.794 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-07T10:57:41.794 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T10:57:41.794 INFO:teuthology.orchestra.run.smithi031.stdout:Install 28 Packages 2023-09-07T10:57:41.794 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrade 2 Packages 2023-09-07T10:57:41.795 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:57:41.795 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 53 M 2023-09-07T10:57:41.796 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-07T10:57:45.699 INFO:teuthology.orchestra.run.smithi031.stdout:(1/30): ceph-radosgw-18.0.0-5797.g7075f693.el9. 2.2 MB/s | 8.2 MB 00:03 2023-09-07T10:57:45.766 INFO:teuthology.orchestra.run.smithi031.stdout:(2/30): ceph-selinux-18.0.0-5797.g7075f693.el9. 363 kB/s | 24 kB 00:00 2023-09-07T10:57:45.975 INFO:teuthology.orchestra.run.smithi031.stdout:(3/30): libcephfs2-18.0.0-5797.g7075f693.el9.x8 3.2 MB/s | 682 kB 00:00 2023-09-07T10:57:46.086 INFO:teuthology.orchestra.run.smithi031.stdout:(4/30): libradosstriper1-18.0.0-5797.g7075f693. 4.2 MB/s | 469 kB 00:00 2023-09-07T10:57:46.592 INFO:teuthology.orchestra.run.smithi031.stdout:(5/30): librgw2-18.0.0-5797.g7075f693.el9.x86_6 8.7 MB/s | 4.4 MB 00:00 2023-09-07T10:57:46.643 INFO:teuthology.orchestra.run.smithi031.stdout:(6/30): python3-ceph-argparse-18.0.0-5797.g7075 906 kB/s | 45 kB 00:00 2023-09-07T10:57:46.693 INFO:teuthology.orchestra.run.smithi031.stdout:(7/30): python3-ceph-common-18.0.0-5797.g7075f6 2.5 MB/s | 127 kB 00:00 2023-09-07T10:57:46.743 INFO:teuthology.orchestra.run.smithi031.stdout:(8/30): python3-cephfs-18.0.0-5797.g7075f693.el 3.2 MB/s | 161 kB 00:00 2023-09-07T10:57:46.811 INFO:teuthology.orchestra.run.smithi031.stdout:(9/30): python3-rados-18.0.0-5797.g7075f693.el9 4.7 MB/s | 321 kB 00:00 2023-09-07T10:57:46.877 INFO:teuthology.orchestra.run.smithi031.stdout:(10/30): python3-rbd-18.0.0-5797.g7075f693.el9. 4.4 MB/s | 298 kB 00:00 2023-09-07T10:57:46.927 INFO:teuthology.orchestra.run.smithi031.stdout:(11/30): python3-rgw-18.0.0-5797.g7075f693.el9. 1.9 MB/s | 99 kB 00:00 2023-09-07T10:57:47.086 INFO:teuthology.orchestra.run.smithi031.stdout:(12/30): boost-program-options-1.75.0-8.el9.x86 669 kB/s | 106 kB 00:00 2023-09-07T10:57:47.170 INFO:teuthology.orchestra.run.smithi031.stdout:(13/30): ceph-base-18.0.0-5797.g7075f693.el9.x8 1.3 MB/s | 6.6 MB 00:05 2023-09-07T10:57:47.195 INFO:teuthology.orchestra.run.smithi031.stdout:(14/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 159 kB 00:00 2023-09-07T10:57:47.278 INFO:teuthology.orchestra.run.smithi031.stdout:(15/30): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 661 kB 00:00 2023-09-07T10:57:47.304 INFO:teuthology.orchestra.run.smithi031.stdout:(16/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.5 MB/s | 282 kB 00:00 2023-09-07T10:57:47.329 INFO:teuthology.orchestra.run.smithi031.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 829 kB/s | 41 kB 00:00 2023-09-07T10:57:47.521 INFO:teuthology.orchestra.run.smithi031.stdout:(18/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 209 kB/s | 45 kB 00:00 2023-09-07T10:57:47.588 INFO:teuthology.orchestra.run.smithi031.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 429 kB/s | 111 kB 00:00 2023-09-07T10:57:47.738 INFO:teuthology.orchestra.run.smithi031.stdout:(20/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.4 MB/s | 309 kB 00:00 2023-09-07T10:57:47.813 INFO:teuthology.orchestra.run.smithi031.stdout:(21/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 325 kB/s | 24 kB 00:00 2023-09-07T10:57:47.872 INFO:teuthology.orchestra.run.smithi031.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 834 kB/s | 49 kB 00:00 2023-09-07T10:57:47.957 INFO:teuthology.orchestra.run.smithi031.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 809 kB/s | 67 kB 00:00 2023-09-07T10:57:48.356 INFO:teuthology.orchestra.run.smithi031.stdout:(24/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 2.0 MB/s | 837 kB 00:00 2023-09-07T10:57:48.490 INFO:teuthology.orchestra.run.smithi031.stdout:(25/30): re2-20211101-2.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2023-09-07T10:57:48.858 INFO:teuthology.orchestra.run.smithi031.stdout:(26/30): thrift-0.15.0-2.el9.x86_64.rpm 4.3 MB/s | 1.6 MB 00:00 2023-09-07T10:57:48.931 INFO:teuthology.orchestra.run.smithi031.stdout:(27/30): libarrow-9.0.0-9.el9.x86_64.rpm 3.3 MB/s | 4.4 MB 00:01 2023-09-07T10:57:49.340 INFO:teuthology.orchestra.run.smithi031.stdout:(28/30): librbd1-18.0.0-5797.g7075f693.el9.x86_ 7.4 MB/s | 3.0 MB 00:00 2023-09-07T10:57:50.023 INFO:teuthology.orchestra.run.smithi031.stdout:(29/30): librados2-18.0.0-5797.g7075f693.el9.x8 2.6 MB/s | 3.1 MB 00:01 2023-09-07T10:57:51.148 INFO:teuthology.orchestra.run.smithi031.stdout:(30/30): ceph-common-18.0.0-5797.g7075f693.el9. 1.9 MB/s | 17 MB 00:09 2023-09-07T10:57:51.152 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-07T10:57:51.153 INFO:teuthology.orchestra.run.smithi031.stdout:Total 5.7 MB/s | 53 MB 00:09 2023-09-07T10:57:51.171 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-07T10:57:51.214 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-07T10:57:51.215 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-07T10:57:51.782 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-07T10:57:51.783 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-07T10:57:53.160 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-07T10:57:53.624 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-09-07T10:57:53.659 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librados2-2:18.0.0-5797.g7075f693.el9.x86_64 2/32 2023-09-07T10:57:53.751 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:18.0.0-5797.g7075f693.el9.x86_64 2/32 2023-09-07T10:57:53.846 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rados-2:18.0.0-5797.g7075f693.el9.x86_64 3/32 2023-09-07T10:57:53.877 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs2-2:18.0.0-5797.g7075f693.el9.x86_64 4/32 2023-09-07T10:57:53.923 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs2-2:18.0.0-5797.g7075f693.el9.x86_64 4/32 2023-09-07T10:57:53.968 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/32 2023-09-07T10:57:54.031 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/32 2023-09-07T10:57:54.069 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/32 2023-09-07T10:57:54.113 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-argparse-2:18.0.0-5797.g7075f693.el9. 8/32 2023-09-07T10:57:54.182 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cephfs-2:18.0.0-5797.g7075f693.el9.x86_64 9/32 2023-09-07T10:57:54.205 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libradosstriper1-2:18.0.0-5797.g7075f693.el9.x86_6 10/32 2023-09-07T10:57:54.301 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5797.g7075f693.el9.x86_6 10/32 2023-09-07T10:57:54.345 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/32 2023-09-07T10:57:54.455 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : re2-1:20211101-2.el9.x86_64 12/32 2023-09-07T10:57:54.497 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-09-07T10:57:54.546 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-09-07T10:57:54.577 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-09-07T10:57:54.605 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-09-07T10:57:54.641 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-09-07T10:57:54.673 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-09-07T10:57:55.010 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-09-07T10:57:55.036 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librbd1-2:18.0.0-5797.g7075f693.el9.x86_64 20/32 2023-09-07T10:57:55.127 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:18.0.0-5797.g7075f693.el9.x86_64 20/32 2023-09-07T10:57:55.158 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rbd-2:18.0.0-5797.g7075f693.el9.x86_64 21/32 2023-09-07T10:57:55.194 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-09-07T10:57:55.640 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-09-07T10:57:55.664 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librgw2-2:18.0.0-5797.g7075f693.el9.x86_64 24/32 2023-09-07T10:57:55.744 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librgw2-2:18.0.0-5797.g7075f693.el9.x86_64 24/32 2023-09-07T10:57:55.823 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rgw-2:18.0.0-5797.g7075f693.el9.x86_64 25/32 2023-09-07T10:57:55.858 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-common-2:18.0.0-5797.g7075f693.el9.x8 26/32 2023-09-07T10:57:58.179 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:18.0.0-5797.g7075f693.el9.x86_64 27/32 2023-09-07T10:57:58.212 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-common-2:18.0.0-5797.g7075f693.el9.x86_64 27/32 2023-09-07T10:57:58.991 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:18.0.0-5797.g7075f693.el9.x86_64 27/32 2023-09-07T10:57:59.032 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-base-2:18.0.0-5797.g7075f693.el9.x86_64 28/32 2023-09-07T10:57:59.121 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-base-2:18.0.0-5797.g7075f693.el9.x86_64 28/32 2023-09-07T10:57:59.121 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-09-07T10:57:59.122 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-09-07T10:57:59.122 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:57:59.149 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-selinux-2:18.0.0-5797.g7075f693.el9.x86_64 29/32 2023-09-07T10:58:09.183 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5797.g7075f693.el9.x86_64 29/32 2023-09-07T10:58:09.183 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /sys 2023-09-07T10:58:09.183 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /proc 2023-09-07T10:58:09.184 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /mnt 2023-09-07T10:58:09.184 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /var/tmp 2023-09-07T10:58:09.184 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /home 2023-09-07T10:58:09.184 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /root 2023-09-07T10:58:09.184 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /tmp 2023-09-07T10:58:09.185 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:58:10.030 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-radosgw-2:18.0.0-5797.g7075f693.el9.x86_64 30/32 2023-09-07T10:58:10.056 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5797.g7075f693.el9.x86_64 30/32 2023-09-07T10:58:10.057 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-09-07T10:58:10.057 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-09-07T10:58:10.057 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-09-07T10:58:10.057 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-09-07T10:58:10.058 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:58:10.058 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-09-07T10:58:10.141 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-09-07T10:58:10.142 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-09-07T10:58:11.284 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-09-07T10:58:11.284 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-base-2:18.0.0-5797.g7075f693.el9.x86_64 1/32 2023-09-07T10:58:11.285 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-common-2:18.0.0-5797.g7075f693.el9.x86_64 2/32 2023-09-07T10:58:11.285 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-radosgw-2:18.0.0-5797.g7075f693.el9.x86_64 3/32 2023-09-07T10:58:11.285 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-selinux-2:18.0.0-5797.g7075f693.el9.x86_64 4/32 2023-09-07T10:58:11.285 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs2-2:18.0.0-5797.g7075f693.el9.x86_64 5/32 2023-09-07T10:58:11.285 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libradosstriper1-2:18.0.0-5797.g7075f693.el9.x86_6 6/32 2023-09-07T10:58:11.286 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librgw2-2:18.0.0-5797.g7075f693.el9.x86_64 7/32 2023-09-07T10:58:11.288 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5797.g7075f693.el9. 8/32 2023-09-07T10:58:11.289 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-common-2:18.0.0-5797.g7075f693.el9.x8 9/32 2023-09-07T10:58:11.289 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cephfs-2:18.0.0-5797.g7075f693.el9.x86_64 10/32 2023-09-07T10:58:11.289 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rados-2:18.0.0-5797.g7075f693.el9.x86_64 11/32 2023-09-07T10:58:11.289 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rbd-2:18.0.0-5797.g7075f693.el9.x86_64 12/32 2023-09-07T10:58:11.290 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rgw-2:18.0.0-5797.g7075f693.el9.x86_64 13/32 2023-09-07T10:58:11.290 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-09-07T10:58:11.290 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-09-07T10:58:11.291 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-09-07T10:58:11.291 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-09-07T10:58:11.291 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-09-07T10:58:11.291 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-09-07T10:58:11.291 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-09-07T10:58:11.292 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-09-07T10:58:11.292 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-09-07T10:58:11.292 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-09-07T10:58:11.292 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-09-07T10:58:11.292 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-09-07T10:58:11.293 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-09-07T10:58:11.293 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : re2-1:20211101-2.el9.x86_64 27/32 2023-09-07T10:58:11.293 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-09-07T10:58:11.293 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:18.0.0-5797.g7075f693.el9.x86_64 29/32 2023-09-07T10:58:11.294 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-09-07T10:58:11.294 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:18.0.0-5797.g7075f693.el9.x86_64 31/32 2023-09-07T10:58:12.286 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-09-07T10:58:12.286 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:58:12.287 INFO:teuthology.orchestra.run.smithi031.stdout:Upgraded: 2023-09-07T10:58:12.287 INFO:teuthology.orchestra.run.smithi031.stdout: librados2-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T10:58:12.287 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T10:58:12.287 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-07T10:58:12.287 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-09-07T10:58:12.287 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T10:58:12.288 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T10:58:12.288 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T10:58:12.288 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T10:58:12.288 INFO:teuthology.orchestra.run.smithi031.stdout: fmt-8.1.1-5.el9.x86_64 2023-09-07T10:58:12.288 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-09-07T10:58:12.289 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-9.0.0-9.el9.x86_64 2023-09-07T10:58:12.289 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-09-07T10:58:12.289 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T10:58:12.289 INFO:teuthology.orchestra.run.smithi031.stdout: liboath-2.6.7-2.el9.x86_64 2023-09-07T10:58:12.289 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-09-07T10:58:12.290 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-09-07T10:58:12.290 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T10:58:12.290 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-09-07T10:58:12.290 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T10:58:12.290 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind-1.6.2-1.el9.x86_64 2023-09-07T10:58:12.291 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-09-07T10:58:12.291 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-09-07T10:58:12.291 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T10:58:12.291 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T10:58:12.291 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T10:58:12.292 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-09-07T10:58:12.292 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T10:58:12.292 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T10:58:12.292 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T10:58:12.292 INFO:teuthology.orchestra.run.smithi031.stdout: re2-1:20211101-2.el9.x86_64 2023-09-07T10:58:12.293 INFO:teuthology.orchestra.run.smithi031.stdout: thrift-0.15.0-2.el9.x86_64 2023-09-07T10:58:12.293 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:58:12.293 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T10:58:12.620 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-test 2023-09-07T10:58:13.178 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:34 ago on Thu 07 Sep 2023 10:57:39 AM UTC. 2023-09-07T10:58:13.328 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T10:58:13.345 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T10:58:13.345 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-09-07T10:58:13.345 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T10:58:13.346 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-07T10:58:13.346 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test x86_64 2:18.0.0-5797.g7075f693.el9 ceph 116 M 2023-09-07T10:58:13.346 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-09-07T10:58:13.346 INFO:teuthology.orchestra.run.smithi031.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2023-09-07T10:58:13.346 INFO:teuthology.orchestra.run.smithi031.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-09-07T10:58:13.347 INFO:teuthology.orchestra.run.smithi031.stdout: libasan x86_64 11.4.1-2.1.el9 appstream 421 k 2023-09-07T10:58:13.347 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite x86_64 2:18.0.0-5797.g7075f693.el9 ceph 161 k 2023-09-07T10:58:13.347 INFO:teuthology.orchestra.run.smithi031.stdout: libubsan x86_64 11.4.1-2.1.el9 appstream 190 k 2023-09-07T10:58:13.347 INFO:teuthology.orchestra.run.smithi031.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-09-07T10:58:13.348 INFO:teuthology.orchestra.run.smithi031.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-09-07T10:58:13.348 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-09-07T10:58:13.348 INFO:teuthology.orchestra.run.smithi031.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2023-09-07T10:58:13.348 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:58:13.348 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-07T10:58:13.349 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T10:58:13.349 INFO:teuthology.orchestra.run.smithi031.stdout:Install 10 Packages 2023-09-07T10:58:13.349 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:58:13.349 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 119 M 2023-09-07T10:58:13.349 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 1.1 G 2023-09-07T10:58:13.350 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-07T10:58:13.706 INFO:teuthology.orchestra.run.smithi031.stdout:(1/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2023-09-07T10:58:13.747 INFO:teuthology.orchestra.run.smithi031.stdout:(2/10): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-09-07T10:58:13.781 INFO:teuthology.orchestra.run.smithi031.stdout:(3/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2023-09-07T10:58:14.239 INFO:teuthology.orchestra.run.smithi031.stdout:(4/10): libcephsqlite-18.0.0-5797.g7075f693.el9 236 kB/s | 161 kB 00:00 2023-09-07T10:58:14.265 INFO:teuthology.orchestra.run.smithi031.stdout:(5/10): jq-1.6-15.el9.x86_64.rpm 387 kB/s | 187 kB 00:00 2023-09-07T10:58:14.482 INFO:teuthology.orchestra.run.smithi031.stdout:(6/10): libubsan-11.4.1-2.1.el9.x86_64.rpm 878 kB/s | 190 kB 00:00 2023-09-07T10:58:14.890 INFO:teuthology.orchestra.run.smithi031.stdout:(7/10): libasan-11.4.1-2.1.el9.x86_64.rpm 648 kB/s | 421 kB 00:00 2023-09-07T10:58:15.007 INFO:teuthology.orchestra.run.smithi031.stdout:(8/10): cryptopp-8.6.0-1.el9.x86_64.rpm 2.4 MB/s | 1.3 MB 00:00 2023-09-07T10:58:15.233 INFO:teuthology.orchestra.run.smithi031.stdout:(9/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 356 kB/s | 122 kB 00:00 2023-09-07T10:58:22.691 INFO:teuthology.orchestra.run.smithi031.stdout:(10/10): ceph-test-18.0.0-5797.g7075f693.el9.x8 13 MB/s | 116 MB 00:09 2023-09-07T10:58:22.695 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-07T10:58:22.695 INFO:teuthology.orchestra.run.smithi031.stdout:Total 13 MB/s | 119 MB 00:09 2023-09-07T10:58:22.735 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-07T10:58:22.759 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-07T10:58:22.759 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-07T10:58:23.606 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-07T10:58:23.607 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-07T10:58:24.738 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-07T10:58:24.833 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2023-09-07T10:58:24.862 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2023-09-07T10:58:24.891 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libubsan-11.4.1-2.1.el9.x86_64 3/10 2023-09-07T10:58:24.934 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libasan-11.4.1-2.1.el9.x86_64 4/10 2023-09-07T10:58:24.991 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/10 2023-09-07T10:58:25.030 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/10 2023-09-07T10:58:25.059 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 7/10 2023-09-07T10:58:25.102 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : jq-1.6-15.el9.x86_64 8/10 2023-09-07T10:58:25.130 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephsqlite-2:18.0.0-5797.g7075f693.el9.x86_64 9/10 2023-09-07T10:58:42.172 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5797.g7075f693.el9.x86_64 9/10 2023-09-07T10:58:42.246 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-test-2:18.0.0-5797.g7075f693.el9.x86_64 10/10 2023-09-07T10:58:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-test-2:18.0.0-5797.g7075f693.el9.x86_64 10/10 2023-09-07T10:58:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-test-2:18.0.0-5797.g7075f693.el9.x86_64 1/10 2023-09-07T10:58:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephsqlite-2:18.0.0-5797.g7075f693.el9.x86_64 2/10 2023-09-07T10:58:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/10 2023-09-07T10:58:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/10 2023-09-07T10:58:49.952 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/10 2023-09-07T10:58:49.952 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : jq-1.6-15.el9.x86_64 6/10 2023-09-07T10:58:49.952 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libasan-11.4.1-2.1.el9.x86_64 7/10 2023-09-07T10:58:49.952 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libubsan-11.4.1-2.1.el9.x86_64 8/10 2023-09-07T10:58:49.952 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2023-09-07T10:58:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2023-09-07T10:58:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:58:50.409 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-07T10:58:50.409 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T10:58:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: cryptopp-8.6.0-1.el9.x86_64 2023-09-07T10:58:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: jq-1.6-15.el9.x86_64 2023-09-07T10:58:50.410 INFO:teuthology.orchestra.run.smithi031.stdout: libasan-11.4.1-2.1.el9.x86_64 2023-09-07T10:58:50.411 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T10:58:50.412 INFO:teuthology.orchestra.run.smithi031.stdout: libubsan-11.4.1-2.1.el9.x86_64 2023-09-07T10:58:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-09-07T10:58:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: socat-1.7.4.1-5.el9.x86_64 2023-09-07T10:58:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-09-07T10:58:50.413 INFO:teuthology.orchestra.run.smithi031.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2023-09-07T10:58:50.414 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:58:50.415 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T10:58:50.707 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph 2023-09-07T10:58:51.267 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:12 ago on Thu 07 Sep 2023 10:57:39 AM UTC. 2023-09-07T10:58:51.419 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T10:58:51.420 INFO:teuthology.orchestra.run.smithi031.stdout:========================================================================================= 2023-09-07T10:58:51.420 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-09-07T10:58:51.421 INFO:teuthology.orchestra.run.smithi031.stdout:========================================================================================= 2023-09-07T10:58:51.421 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-07T10:58:51.421 INFO:teuthology.orchestra.run.smithi031.stdout: ceph x86_64 2:18.0.0-5797.g7075f693.el9 ceph 6.5 k 2023-09-07T10:58:51.422 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-09-07T10:58:51.422 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds x86_64 2:18.0.0-5797.g7075f693.el9 ceph 2.0 M 2023-09-07T10:58:51.422 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr x86_64 2:18.0.0-5797.g7075f693.el9 ceph 1.4 M 2023-09-07T10:58:51.422 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5797.g7075f693.el9 ceph-noarch 242 k 2023-09-07T10:58:51.422 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon x86_64 2:18.0.0-5797.g7075f693.el9 ceph 8.8 M 2023-09-07T10:58:51.423 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd x86_64 2:18.0.0-5797.g7075f693.el9 ceph 116 M 2023-09-07T10:58:51.423 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-09-07T10:58:51.423 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-09-07T10:58:51.424 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-09-07T10:58:51.424 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-09-07T10:58:51.424 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-09-07T10:58:51.425 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-09-07T10:58:51.425 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-09-07T10:58:51.425 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-09-07T10:58:51.425 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-09-07T10:58:51.425 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-09-07T10:58:51.426 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-09-07T10:58:51.426 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-09-07T10:58:51.426 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-09-07T10:58:51.426 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-09-07T10:58:51.427 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-09-07T10:58:51.427 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-09-07T10:58:51.427 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-09-07T10:58:51.427 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-09-07T10:58:51.427 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-09-07T10:58:51.428 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-09-07T10:58:51.428 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-09-07T10:58:51.428 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-09-07T10:58:51.428 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-09-07T10:58:51.428 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3 noarch 1.26.5-3.el9 CentOS-BaseOS 188 k 2023-09-07T10:58:51.429 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-09-07T10:58:51.429 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-09-07T10:58:51.429 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-09-07T10:58:51.429 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:58:51.430 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-07T10:58:51.430 INFO:teuthology.orchestra.run.smithi031.stdout:========================================================================================= 2023-09-07T10:58:51.430 INFO:teuthology.orchestra.run.smithi031.stdout:Install 33 Packages 2023-09-07T10:58:51.431 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:58:51.432 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 133 M 2023-09-07T10:58:51.432 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 1.4 G 2023-09-07T10:58:51.432 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-07T10:58:52.101 INFO:teuthology.orchestra.run.smithi031.stdout:(1/33): ceph-18.0.0-5797.g7075f693.el9.x86_64.r 15 kB/s | 6.5 kB 00:00 2023-09-07T10:58:52.668 INFO:teuthology.orchestra.run.smithi031.stdout:(2/33): ceph-mgr-18.0.0-5797.g7075f693.el9.x86_ 1.4 MB/s | 1.4 MB 00:01 2023-09-07T10:58:53.936 INFO:teuthology.orchestra.run.smithi031.stdout:(3/33): ceph-mds-18.0.0-5797.g7075f693.el9.x86_ 912 kB/s | 2.0 MB 00:02 2023-09-07T10:58:54.068 INFO:teuthology.orchestra.run.smithi031.stdout:(4/33): ceph-mgr-modules-core-18.0.0-5797.g7075 1.8 MB/s | 242 kB 00:00 2023-09-07T10:58:54.193 INFO:teuthology.orchestra.run.smithi031.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2023-09-07T10:58:54.229 INFO:teuthology.orchestra.run.smithi031.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 944 kB/s | 32 kB 00:00 2023-09-07T10:58:54.264 INFO:teuthology.orchestra.run.smithi031.stdout:(7/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-09-07T10:58:54.303 INFO:teuthology.orchestra.run.smithi031.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2023-09-07T10:58:54.338 INFO:teuthology.orchestra.run.smithi031.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-09-07T10:58:54.379 INFO:teuthology.orchestra.run.smithi031.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2023-09-07T10:58:54.412 INFO:teuthology.orchestra.run.smithi031.stdout:(11/33): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-09-07T10:58:54.454 INFO:teuthology.orchestra.run.smithi031.stdout:(12/33): python3-urllib3-1.26.5-3.el9.noarch.rp 4.4 MB/s | 188 kB 00:00 2023-09-07T10:58:55.298 INFO:teuthology.orchestra.run.smithi031.stdout:(13/33): python3-cryptography-36.0.1-4.el9.x86_ 1.5 MB/s | 1.2 MB 00:00 2023-09-07T10:58:55.505 INFO:teuthology.orchestra.run.smithi031.stdout:(14/33): ceph-mon-18.0.0-5797.g7075f693.el9.x86 2.6 MB/s | 8.8 MB 00:03 2023-09-07T10:58:55.663 INFO:teuthology.orchestra.run.smithi031.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 139 kB/s | 51 kB 00:00 2023-09-07T10:58:55.805 INFO:teuthology.orchestra.run.smithi031.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 145 kB/s | 43 kB 00:00 2023-09-07T10:58:56.047 INFO:teuthology.orchestra.run.smithi031.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2023-09-07T10:58:56.073 INFO:teuthology.orchestra.run.smithi031.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 420 kB/s | 172 kB 00:00 2023-09-07T10:58:56.131 INFO:teuthology.orchestra.run.smithi031.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2023-09-07T10:58:56.165 INFO:teuthology.orchestra.run.smithi031.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2023-09-07T10:58:56.223 INFO:teuthology.orchestra.run.smithi031.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 251 kB/s | 23 kB 00:00 2023-09-07T10:58:56.257 INFO:teuthology.orchestra.run.smithi031.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 211 kB/s | 19 kB 00:00 2023-09-07T10:58:56.316 INFO:teuthology.orchestra.run.smithi031.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2023-09-07T10:58:56.357 INFO:teuthology.orchestra.run.smithi031.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 462 kB/s | 46 kB 00:00 2023-09-07T10:58:56.408 INFO:teuthology.orchestra.run.smithi031.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 857 kB/s | 79 kB 00:00 2023-09-07T10:58:56.500 INFO:teuthology.orchestra.run.smithi031.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 179 kB/s | 16 kB 00:00 2023-09-07T10:58:56.534 INFO:teuthology.orchestra.run.smithi031.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2023-09-07T10:58:56.592 INFO:teuthology.orchestra.run.smithi031.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 979 kB/s | 90 kB 00:00 2023-09-07T10:58:56.638 INFO:teuthology.orchestra.run.smithi031.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 357 kB/s | 36 kB 00:00 2023-09-07T10:58:56.685 INFO:teuthology.orchestra.run.smithi031.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2023-09-07T10:58:56.735 INFO:teuthology.orchestra.run.smithi031.stdout:(31/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2023-09-07T10:58:56.777 INFO:teuthology.orchestra.run.smithi031.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 217 kB/s | 20 kB 00:00 2023-09-07T10:59:15.045 INFO:teuthology.orchestra.run.smithi031.stdout:(33/33): ceph-osd-18.0.0-5797.g7075f693.el9.x86 5.2 MB/s | 116 MB 00:22 2023-09-07T10:59:15.045 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-07T10:59:15.046 INFO:teuthology.orchestra.run.smithi031.stdout:Total 5.6 MB/s | 133 MB 00:23 2023-09-07T10:59:15.079 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-07T10:59:15.124 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-07T10:59:15.124 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-07T10:59:16.168 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-07T10:59:16.168 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-07T10:59:18.030 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-07T10:59:18.102 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-09-07T10:59:18.135 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-09-07T10:59:18.168 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-09-07T10:59:18.264 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-09-07T10:59:18.342 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-09-07T10:59:18.396 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-09-07T10:59:18.434 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-09-07T10:59:18.514 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-09-07T10:59:18.573 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-urllib3-1.26.5-3.el9.noarch 9/33 2023-09-07T10:59:18.610 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-09-07T10:59:18.669 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-09-07T10:59:18.737 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-09-07T10:59:18.947 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-09-07T10:59:18.998 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-09-07T10:59:19.043 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-09-07T10:59:19.080 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-09-07T10:59:19.109 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-09-07T10:59:19.196 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-09-07T10:59:19.228 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-09-07T10:59:19.270 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-09-07T10:59:19.312 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-09-07T10:59:19.346 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-09-07T10:59:19.441 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-09-07T10:59:19.485 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-09-07T10:59:19.562 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-09-07T10:59:19.705 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-09-07T10:59:19.836 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-09-07T10:59:20.021 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5797.g7075f693.el9. 28/33 2023-09-07T10:59:20.050 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-2:18.0.0-5797.g7075f693.el9.x86_64 29/33 2023-09-07T10:59:20.072 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5797.g7075f693.el9.x86_64 29/33 2023-09-07T10:59:20.073 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-09-07T10:59:20.073 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-09-07T10:59:20.073 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-09-07T10:59:20.074 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-09-07T10:59:20.074 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:59:39.230 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-osd-2:18.0.0-5797.g7075f693.el9.x86_64 30/33 2023-09-07T10:59:39.257 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-osd-2:18.0.0-5797.g7075f693.el9.x86_64 30/33 2023-09-07T10:59:39.257 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-09-07T10:59:39.257 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-09-07T10:59:39.258 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-09-07T10:59:39.258 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-09-07T10:59:39.258 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:59:40.189 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mon-2:18.0.0-5797.g7075f693.el9.x86_64 31/33 2023-09-07T10:59:40.213 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mon-2:18.0.0-5797.g7075f693.el9.x86_64 31/33 2023-09-07T10:59:40.214 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-09-07T10:59:40.214 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-09-07T10:59:40.214 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-09-07T10:59:40.214 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-09-07T10:59:40.215 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:59:40.423 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mds-2:18.0.0-5797.g7075f693.el9.x86_64 32/33 2023-09-07T10:59:40.447 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mds-2:18.0.0-5797.g7075f693.el9.x86_64 32/33 2023-09-07T10:59:40.447 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-09-07T10:59:40.447 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-09-07T10:59:40.448 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-09-07T10:59:40.448 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-09-07T10:59:40.448 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:59:40.514 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-2:18.0.0-5797.g7075f693.el9.x86_64 33/33 2023-09-07T10:59:50.113 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-2:18.0.0-5797.g7075f693.el9.x86_64 33/33 2023-09-07T10:59:50.114 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-2:18.0.0-5797.g7075f693.el9.x86_64 1/33 2023-09-07T10:59:50.114 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mds-2:18.0.0-5797.g7075f693.el9.x86_64 2/33 2023-09-07T10:59:50.114 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-2:18.0.0-5797.g7075f693.el9.x86_64 3/33 2023-09-07T10:59:50.114 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mon-2:18.0.0-5797.g7075f693.el9.x86_64 4/33 2023-09-07T10:59:50.115 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-osd-2:18.0.0-5797.g7075f693.el9.x86_64 5/33 2023-09-07T10:59:50.115 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5797.g7075f693.el9. 6/33 2023-09-07T10:59:50.116 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-09-07T10:59:50.116 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-09-07T10:59:50.117 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-09-07T10:59:50.117 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-09-07T10:59:50.117 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-09-07T10:59:50.117 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-09-07T10:59:50.117 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-09-07T10:59:50.118 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-urllib3-1.26.5-3.el9.noarch 14/33 2023-09-07T10:59:50.118 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-09-07T10:59:50.118 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-09-07T10:59:50.118 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-09-07T10:59:50.119 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-09-07T10:59:50.119 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-09-07T10:59:50.119 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-09-07T10:59:50.119 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-09-07T10:59:50.119 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-09-07T10:59:50.120 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-09-07T10:59:50.120 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-09-07T10:59:50.120 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-09-07T10:59:50.120 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-09-07T10:59:50.120 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-09-07T10:59:50.121 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-09-07T10:59:50.121 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-09-07T10:59:50.121 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-09-07T10:59:50.121 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-09-07T10:59:50.121 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-09-07T10:59:51.199 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-09-07T10:59:51.199 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:59:51.199 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-07T10:59:51.200 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T10:59:51.200 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T10:59:51.200 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T10:59:51.200 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core-2:18.0.0-5797.g7075f693.el9.noarch 2023-09-07T10:59:51.200 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T10:59:51.201 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T10:59:51.201 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-09-07T10:59:51.201 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-09-07T10:59:51.201 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-09-07T10:59:51.201 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-09-07T10:59:51.202 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-09-07T10:59:51.202 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-09-07T10:59:51.202 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-09-07T10:59:51.202 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-09-07T10:59:51.202 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-09-07T10:59:51.203 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-09-07T10:59:51.203 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-09-07T10:59:51.203 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako-1.1.4-6.el9.noarch 2023-09-07T10:59:51.203 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-09-07T10:59:51.203 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-09-07T10:59:51.204 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-09-07T10:59:51.204 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply-3.11-14.el9.noarch 2023-09-07T10:59:51.204 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend-3.1.0-2.el9.noarch 2023-09-07T10:59:51.204 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-09-07T10:59:51.204 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser-2.20-6.el9.noarch 2023-09-07T10:59:51.205 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz-2021.1-5.el9.noarch 2023-09-07T10:59:51.205 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-2.25.1-7.el9.noarch 2023-09-07T10:59:51.205 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-09-07T10:59:51.205 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml-0.10.2-6.el9.noarch 2023-09-07T10:59:51.205 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3-1.26.5-3.el9.noarch 2023-09-07T10:59:51.205 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob-1.8.7-6.el9.noarch 2023-09-07T10:59:51.206 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-09-07T10:59:51.207 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-09-07T10:59:51.207 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:59:51.207 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T10:59:51.439 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-base 2023-09-07T10:59:52.012 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:13 ago on Thu 07 Sep 2023 10:57:39 AM UTC. 2023-09-07T10:59:52.104 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-base-2:18.0.0-5797.g7075f693.el9.x86_64 is already installed. 2023-09-07T10:59:52.155 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T10:59:52.156 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-09-07T10:59:52.156 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T10:59:52.214 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install cephadm 2023-09-07T10:59:52.774 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:13 ago on Thu 07 Sep 2023 10:57:39 AM UTC. 2023-09-07T10:59:52.915 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T10:59:52.916 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T10:59:52.916 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-09-07T10:59:52.916 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T10:59:52.917 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-07T10:59:52.917 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm noarch 2:18.0.0-5797.g7075f693.el9 ceph-noarch 225 k 2023-09-07T10:59:52.917 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:59:52.917 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-07T10:59:52.918 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T10:59:52.918 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-09-07T10:59:52.918 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:59:52.918 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 225 k 2023-09-07T10:59:52.918 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 217 k 2023-09-07T10:59:52.919 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-07T10:59:53.139 INFO:teuthology.orchestra.run.smithi031.stdout:cephadm-18.0.0-5797.g7075f693.el9.noarch.rpm 1.0 MB/s | 225 kB 00:00 2023-09-07T10:59:53.140 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-07T10:59:53.140 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.0 MB/s | 225 kB 00:00 2023-09-07T10:59:53.140 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-07T10:59:53.146 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-07T10:59:53.146 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-07T10:59:53.148 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-07T10:59:53.149 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-07T10:59:53.221 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-07T10:59:53.659 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:18.0.0-5797.g7075f693.el9.noarch 1/1 2023-09-07T10:59:53.738 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephadm-2:18.0.0-5797.g7075f693.el9.noarch 1/1 2023-09-07T10:59:54.108 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:18.0.0-5797.g7075f693.el9.noarch 1/1 2023-09-07T10:59:54.407 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephadm-2:18.0.0-5797.g7075f693.el9.noarch 1/1 2023-09-07T10:59:54.408 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:59:54.408 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-07T10:59:54.408 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm-2:18.0.0-5797.g7075f693.el9.noarch 2023-09-07T10:59:54.408 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:59:54.409 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T10:59:54.616 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-immutable-object-cache 2023-09-07T10:59:55.176 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:16 ago on Thu 07 Sep 2023 10:57:39 AM UTC. 2023-09-07T10:59:55.317 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T10:59:55.318 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T10:59:55.319 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2023-09-07T10:59:55.319 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T10:59:55.319 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-07T10:59:55.319 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5797.g7075f693.el9 ceph 144 k 2023-09-07T10:59:55.319 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:59:55.320 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-07T10:59:55.320 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T10:59:55.320 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-09-07T10:59:55.320 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:59:55.320 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 144 k 2023-09-07T10:59:55.321 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 437 k 2023-09-07T10:59:55.321 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-07T10:59:55.628 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-immutable-object-cache-18.0.0-5797.g7075f6 466 kB/s | 144 kB 00:00 2023-09-07T10:59:55.629 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-07T10:59:55.629 INFO:teuthology.orchestra.run.smithi031.stdout:Total 464 kB/s | 144 kB 00:00 2023-09-07T10:59:55.630 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-07T10:59:55.638 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-07T10:59:55.638 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-07T10:59:55.693 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-07T10:59:55.693 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-07T10:59:55.832 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-07T10:59:55.906 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5797.g7075f693. 1/1 2023-09-07T10:59:55.930 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5797.g7075f693. 1/1 2023-09-07T10:59:55.930 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-09-07T10:59:55.931 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-09-07T10:59:55.931 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:59:56.943 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5797.g7075f693. 1/1 2023-09-07T10:59:56.944 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:59:56.944 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-07T10:59:56.944 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T10:59:56.944 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:59:56.945 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T10:59:57.161 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr 2023-09-07T10:59:57.726 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:18 ago on Thu 07 Sep 2023 10:57:39 AM UTC. 2023-09-07T10:59:57.816 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-mgr-2:18.0.0-5797.g7075f693.el9.x86_64 is already installed. 2023-09-07T10:59:57.868 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T10:59:57.868 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-09-07T10:59:57.868 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T10:59:57.933 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-dashboard 2023-09-07T10:59:58.491 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:19 ago on Thu 07 Sep 2023 10:57:39 AM UTC. 2023-09-07T10:59:58.632 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T10:59:58.633 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T10:59:58.633 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-09-07T10:59:58.633 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T10:59:58.633 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-07T10:59:58.634 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5797.g7075f693.el9 ceph-noarch 1.7 M 2023-09-07T10:59:58.634 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-09-07T10:59:58.634 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5797.g7075f693.el9 ceph-noarch 23 k 2023-09-07T10:59:58.634 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5797.g7075f693.el9 ceph-noarch 15 k 2023-09-07T10:59:58.634 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-09-07T10:59:58.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-09-07T10:59:58.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-09-07T10:59:58.635 INFO:teuthology.orchestra.run.smithi031.stdout:Installing weak dependencies: 2023-09-07T10:59:58.635 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-09-07T10:59:58.635 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:59:58.636 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-07T10:59:58.636 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T10:59:58.636 INFO:teuthology.orchestra.run.smithi031.stdout:Install 7 Packages 2023-09-07T10:59:58.636 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T10:59:58.636 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 2.0 M 2023-09-07T10:59:58.637 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 8.5 M 2023-09-07T10:59:58.637 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-07T10:59:59.225 INFO:teuthology.orchestra.run.smithi031.stdout:(1/7): ceph-grafana-dashboards-18.0.0-5797.g707 135 kB/s | 23 kB 00:00 2023-09-07T10:59:59.325 INFO:teuthology.orchestra.run.smithi031.stdout:(2/7): ceph-prometheus-alerts-18.0.0-5797.g7075 54 kB/s | 15 kB 00:00 2023-09-07T10:59:59.575 INFO:teuthology.orchestra.run.smithi031.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 26 kB/s | 9.0 kB 00:00 2023-09-07T10:59:59.751 INFO:teuthology.orchestra.run.smithi031.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 176 kB/s | 31 kB 00:00 2023-09-07T10:59:59.860 INFO:teuthology.orchestra.run.smithi031.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 80 kB/s | 41 kB 00:00 2023-09-07T11:00:00.043 INFO:teuthology.orchestra.run.smithi031.stdout:(6/7): ceph-mgr-dashboard-18.0.0-5797.g7075f693 1.8 MB/s | 1.7 MB 00:00 2023-09-07T11:00:00.068 INFO:teuthology.orchestra.run.smithi031.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 594 kB/s | 188 kB 00:00 2023-09-07T11:00:00.072 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-07T11:00:00.072 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2023-09-07T11:00:00.072 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-07T11:00:00.082 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-07T11:00:00.082 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-07T11:00:00.155 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-07T11:00:00.155 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-07T11:00:00.474 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-07T11:00:00.528 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-09-07T11:00:00.562 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-09-07T11:00:00.609 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-09-07T11:00:00.640 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-09-07T11:00:00.669 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5797.g7075f693.el9.n 5/7 2023-09-07T11:00:00.909 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5797.g7075f693.el9. 6/7 2023-09-07T11:00:00.946 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5797.g7075f693.el9.noarc 7/7 2023-09-07T11:00:01.532 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5797.g7075f693.el9.noarc 7/7 2023-09-07T11:00:01.532 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5797.g7075f693.el9. 1/7 2023-09-07T11:00:01.532 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5797.g7075f693.el9.noarc 2/7 2023-09-07T11:00:01.532 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5797.g7075f693.el9.n 3/7 2023-09-07T11:00:01.533 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-09-07T11:00:01.533 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-09-07T11:00:01.533 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-09-07T11:00:01.850 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-09-07T11:00:01.850 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:01.850 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-07T11:00:01.851 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards-2:18.0.0-5797.g7075f693.el9.noarch 2023-09-07T11:00:01.851 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard-2:18.0.0-5797.g7075f693.el9.noarch 2023-09-07T11:00:01.851 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts-2:18.0.0-5797.g7075f693.el9.noarch 2023-09-07T11:00:01.851 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-09-07T11:00:01.851 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-09-07T11:00:01.852 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-09-07T11:00:01.852 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes-2.5.1-5.el9.noarch 2023-09-07T11:00:01.852 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:01.852 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T11:00:02.048 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-07T11:00:02.631 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:23 ago on Thu 07 Sep 2023 10:57:39 AM UTC. 2023-09-07T11:00:02.781 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T11:00:02.783 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2023-09-07T11:00:02.783 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-09-07T11:00:02.783 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2023-09-07T11:00:02.784 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-07T11:00:02.784 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5797.g7075f693.el9 ceph-noarch 7.4 M 2023-09-07T11:00:02.784 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-09-07T11:00:02.784 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-09-07T11:00:02.784 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-09-07T11:00:02.785 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-09-07T11:00:02.785 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran x86_64 11.4.1-2.1.el9 baseos 809 k 2023-09-07T11:00:02.785 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath x86_64 11.4.1-2.1.el9 baseos 198 k 2023-09-07T11:00:02.785 INFO:teuthology.orchestra.run.smithi031.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-09-07T11:00:02.786 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-09-07T11:00:02.786 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-09-07T11:00:02.786 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-09-07T11:00:02.787 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-09-07T11:00:02.787 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-09-07T11:00:02.787 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:02.788 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-07T11:00:02.788 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2023-09-07T11:00:02.788 INFO:teuthology.orchestra.run.smithi031.stdout:Install 12 Packages 2023-09-07T11:00:02.788 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:02.789 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 37 M 2023-09-07T11:00:02.789 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 209 M 2023-09-07T11:00:02.790 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-07T11:00:02.975 INFO:teuthology.orchestra.run.smithi031.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 235 kB/s | 32 kB 00:00 2023-09-07T11:00:03.008 INFO:teuthology.orchestra.run.smithi031.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 533 kB/s | 17 kB 00:00 2023-09-07T11:00:03.044 INFO:teuthology.orchestra.run.smithi031.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2023-09-07T11:00:03.101 INFO:teuthology.orchestra.run.smithi031.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2023-09-07T11:00:03.184 INFO:teuthology.orchestra.run.smithi031.stdout:(5/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.4 MB/s | 205 kB 00:00 2023-09-07T11:00:03.269 INFO:teuthology.orchestra.run.smithi031.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2023-09-07T11:00:03.369 INFO:teuthology.orchestra.run.smithi031.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 26 MB/s | 4.8 MB 00:00 2023-09-07T11:00:03.394 INFO:teuthology.orchestra.run.smithi031.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 217 kB 00:00 2023-09-07T11:00:03.761 INFO:teuthology.orchestra.run.smithi031.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2023-09-07T11:00:04.177 INFO:teuthology.orchestra.run.smithi031.stdout:(10/12): libgfortran-11.4.1-2.1.el9.x86_64.rpm 1.0 MB/s | 809 kB 00:00 2023-09-07T11:00:04.324 INFO:teuthology.orchestra.run.smithi031.stdout:(11/12): libquadmath-11.4.1-2.1.el9.x86_64.rpm 353 kB/s | 198 kB 00:00 2023-09-07T11:00:04.500 INFO:teuthology.orchestra.run.smithi031.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-5 4.5 MB/s | 7.4 MB 00:01 2023-09-07T11:00:04.503 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-07T11:00:04.503 INFO:teuthology.orchestra.run.smithi031.stdout:Total 22 MB/s | 37 MB 00:01 2023-09-07T11:00:04.532 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-07T11:00:04.557 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-07T11:00:04.557 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-07T11:00:05.016 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-07T11:00:05.016 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-07T11:00:05.737 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-07T11:00:05.814 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libquadmath-11.4.1-2.1.el9.x86_64 1/12 2023-09-07T11:00:05.854 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libgfortran-11.4.1-2.1.el9.x86_64 2/12 2023-09-07T11:00:05.978 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-09-07T11:00:06.023 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-09-07T11:00:06.210 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-09-07T11:00:06.238 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-09-07T11:00:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-09-07T11:00:07.040 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-09-07T11:00:07.155 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-09-07T11:00:08.451 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-09-07T11:00:09.517 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-09-07T11:00:09.545 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5797.g7075f 12/12 2023-09-07T11:00:12.091 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5797.g7075f 12/12 2023-09-07T11:00:12.091 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5797.g7075f 1/12 2023-09-07T11:00:12.092 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-09-07T11:00:12.092 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-09-07T11:00:12.092 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-09-07T11:00:12.092 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-09-07T11:00:12.092 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-09-07T11:00:12.093 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-09-07T11:00:12.093 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-09-07T11:00:12.093 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-09-07T11:00:12.093 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-09-07T11:00:12.093 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libgfortran-11.4.1-2.1.el9.x86_64 11/12 2023-09-07T11:00:12.533 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libquadmath-11.4.1-2.1.el9.x86_64 12/12 2023-09-07T11:00:12.534 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:12.534 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-07T11:00:12.534 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5797.g7075f693.el9.noarch 2023-09-07T11:00:12.534 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-09-07T11:00:12.534 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-09-07T11:00:12.535 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-09-07T11:00:12.535 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran-11.4.1-2.1.el9.x86_64 2023-09-07T11:00:12.535 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath-11.4.1-2.1.el9.x86_64 2023-09-07T11:00:12.535 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-0.3.21-2.el9.x86_64 2023-09-07T11:00:12.536 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-09-07T11:00:12.536 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-09-07T11:00:12.536 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-09-07T11:00:12.536 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-09-07T11:00:12.536 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-09-07T11:00:12.537 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:12.537 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T11:00:12.929 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-rook 2023-09-07T11:00:13.490 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:34 ago on Thu 07 Sep 2023 10:57:39 AM UTC. 2023-09-07T11:00:13.634 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T11:00:13.635 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2023-09-07T11:00:13.635 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-09-07T11:00:13.635 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2023-09-07T11:00:13.636 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-07T11:00:13.636 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook noarch 2:18.0.0-5797.g7075f693.el9 ceph-noarch 47 k 2023-09-07T11:00:13.636 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-09-07T11:00:13.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-09-07T11:00:13.636 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-09-07T11:00:13.637 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth noarch 1:2.22.0-1.el9 epel 197 k 2023-09-07T11:00:13.637 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-09-07T11:00:13.637 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-09-07T11:00:13.637 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-09-07T11:00:13.638 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-09-07T11:00:13.638 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-09-07T11:00:13.638 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-09-07T11:00:13.638 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-09-07T11:00:13.639 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-09-07T11:00:13.639 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-09-07T11:00:13.639 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:13.640 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-07T11:00:13.640 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2023-09-07T11:00:13.640 INFO:teuthology.orchestra.run.smithi031.stdout:Install 13 Packages 2023-09-07T11:00:13.640 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:13.641 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 2.0 M 2023-09-07T11:00:13.641 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 26 M 2023-09-07T11:00:13.642 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-07T11:00:13.927 INFO:teuthology.orchestra.run.smithi031.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 207 kB/s | 24 kB 00:00 2023-09-07T11:00:13.944 INFO:teuthology.orchestra.run.smithi031.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 134 kB/s | 18 kB 00:00 2023-09-07T11:00:13.994 INFO:teuthology.orchestra.run.smithi031.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2023-09-07T11:00:14.027 INFO:teuthology.orchestra.run.smithi031.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2023-09-07T11:00:14.053 INFO:teuthology.orchestra.run.smithi031.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2023-09-07T11:00:14.070 INFO:teuthology.orchestra.run.smithi031.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2023-09-07T11:00:14.282 INFO:teuthology.orchestra.run.smithi031.stdout:(7/13): ceph-mgr-rook-18.0.0-5797.g7075f693.el9 101 kB/s | 47 kB 00:00 2023-09-07T11:00:14.463 INFO:teuthology.orchestra.run.smithi031.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 79 kB/s | 32 kB 00:00 2023-09-07T11:00:14.479 INFO:teuthology.orchestra.run.smithi031.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 37 kB/s | 15 kB 00:00 2023-09-07T11:00:14.763 INFO:teuthology.orchestra.run.smithi031.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 209 kB/s | 59 kB 00:00 2023-09-07T11:00:14.930 INFO:teuthology.orchestra.run.smithi031.stdout:(11/13): python3-google-auth-2.22.0-1.el9.noarc 302 kB/s | 197 kB 00:00 2023-09-07T11:00:15.046 INFO:teuthology.orchestra.run.smithi031.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 316 kB/s | 90 kB 00:00 2023-09-07T11:00:15.122 INFO:teuthology.orchestra.run.smithi031.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.5 MB/s | 1.0 MB 00:00 2023-09-07T11:00:15.124 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-07T11:00:15.124 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2023-09-07T11:00:15.125 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-07T11:00:15.142 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-07T11:00:15.142 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-07T11:00:15.255 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-07T11:00:15.255 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-07T11:00:15.792 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-07T11:00:15.970 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-09-07T11:00:16.025 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-09-07T11:00:16.088 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-09-07T11:00:16.124 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-09-07T11:00:16.167 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-09-07T11:00:16.265 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-09-07T11:00:16.383 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-google-auth-1:2.22.0-1.el9.noarch 7/13 2023-09-07T11:00:16.443 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-09-07T11:00:17.210 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-09-07T11:00:17.265 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-09-07T11:00:17.309 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-09-07T11:00:17.361 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-09-07T11:00:17.383 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-rook-2:18.0.0-5797.g7075f693.el9.noarch 13/13 2023-09-07T11:00:18.258 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5797.g7075f693.el9.noarch 13/13 2023-09-07T11:00:18.259 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5797.g7075f693.el9.noarch 1/13 2023-09-07T11:00:18.259 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-09-07T11:00:18.259 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-09-07T11:00:18.259 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-09-07T11:00:18.260 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-09-07T11:00:18.260 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-09-07T11:00:18.260 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-09-07T11:00:18.260 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-09-07T11:00:18.260 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-09-07T11:00:18.260 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-google-auth-1:2.22.0-1.el9.noarch 10/13 2023-09-07T11:00:18.261 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-09-07T11:00:18.261 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-09-07T11:00:18.631 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-09-07T11:00:18.632 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:18.632 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-07T11:00:18.632 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook-2:18.0.0-5797.g7075f693.el9.noarch 2023-09-07T11:00:18.632 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-09-07T11:00:18.632 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-09-07T11:00:18.633 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth-1:2.22.0-1.el9.noarch 2023-09-07T11:00:18.633 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-09-07T11:00:18.633 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-09-07T11:00:18.633 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-09-07T11:00:18.633 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-09-07T11:00:18.634 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-09-07T11:00:18.634 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-09-07T11:00:18.634 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-09-07T11:00:18.634 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa-4.9-2.el9.noarch 2023-09-07T11:00:18.634 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-09-07T11:00:18.635 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:18.635 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T11:00:18.871 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-cephadm 2023-09-07T11:00:19.439 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:40 ago on Thu 07 Sep 2023 10:57:39 AM UTC. 2023-09-07T11:00:19.582 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T11:00:19.583 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2023-09-07T11:00:19.583 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-09-07T11:00:19.583 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2023-09-07T11:00:19.584 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-07T11:00:19.584 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5797.g7075f693.el9 ceph-noarch 132 k 2023-09-07T11:00:19.584 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-09-07T11:00:19.584 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-09-07T11:00:19.584 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-09-07T11:00:19.585 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-09-07T11:00:19.585 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-09-07T11:00:19.585 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-09-07T11:00:19.585 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:19.586 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-07T11:00:19.586 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================== 2023-09-07T11:00:19.587 INFO:teuthology.orchestra.run.smithi031.stdout:Install 6 Packages 2023-09-07T11:00:19.587 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:19.587 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 6.8 M 2023-09-07T11:00:19.588 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 33 M 2023-09-07T11:00:19.588 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-07T11:00:19.964 INFO:teuthology.orchestra.run.smithi031.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.1 MB/s | 228 kB 00:00 2023-09-07T11:00:20.006 INFO:teuthology.orchestra.run.smithi031.stdout:(2/6): ceph-mgr-cephadm-18.0.0-5797.g7075f693.e 541 kB/s | 132 kB 00:00 2023-09-07T11:00:20.340 INFO:teuthology.orchestra.run.smithi031.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 10 MB/s | 5.8 MB 00:00 2023-09-07T11:00:20.525 INFO:teuthology.orchestra.run.smithi031.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 112 kB/s | 58 kB 00:00 2023-09-07T11:00:20.692 INFO:teuthology.orchestra.run.smithi031.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 145 kB/s | 51 kB 00:00 2023-09-07T11:00:20.749 INFO:teuthology.orchestra.run.smithi031.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 699 kB/s | 548 kB 00:00 2023-09-07T11:00:20.752 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-07T11:00:20.753 INFO:teuthology.orchestra.run.smithi031.stdout:Total 5.8 MB/s | 6.8 MB 00:01 2023-09-07T11:00:20.753 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-07T11:00:20.763 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-07T11:00:20.764 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-07T11:00:20.860 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-07T11:00:20.860 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-07T11:00:21.166 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-07T11:00:21.349 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-09-07T11:00:21.402 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-09-07T11:00:21.963 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-09-07T11:00:22.064 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-09-07T11:00:22.132 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-09-07T11:00:22.162 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5797.g7075f693.el9.noarch 6/6 2023-09-07T11:00:22.981 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5797.g7075f693.el9.noarch 6/6 2023-09-07T11:00:22.982 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5797.g7075f693.el9.noarch 1/6 2023-09-07T11:00:22.982 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-09-07T11:00:22.982 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-09-07T11:00:22.982 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-09-07T11:00:22.983 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-09-07T11:00:23.311 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-09-07T11:00:23.312 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:23.312 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-07T11:00:23.312 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm-2:18.0.0-5797.g7075f693.el9.noarch 2023-09-07T11:00:23.312 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-09-07T11:00:23.312 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel-2.9.1-2.el9.noarch 2023-09-07T11:00:23.313 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-09-07T11:00:23.313 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-09-07T11:00:23.313 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-09-07T11:00:23.313 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:23.313 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T11:00:23.549 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-fuse 2023-09-07T11:00:24.131 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:45 ago on Thu 07 Sep 2023 10:57:39 AM UTC. 2023-09-07T11:00:24.275 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T11:00:24.276 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T11:00:24.276 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-09-07T11:00:24.277 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T11:00:24.277 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-07T11:00:24.277 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse x86_64 2:18.0.0-5797.g7075f693.el9 ceph 825 k 2023-09-07T11:00:24.277 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-09-07T11:00:24.278 INFO:teuthology.orchestra.run.smithi031.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-09-07T11:00:24.278 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:24.278 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-07T11:00:24.278 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T11:00:24.278 INFO:teuthology.orchestra.run.smithi031.stdout:Install 2 Packages 2023-09-07T11:00:24.279 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:24.279 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 903 k 2023-09-07T11:00:24.279 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 2.9 M 2023-09-07T11:00:24.279 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-07T11:00:24.414 INFO:teuthology.orchestra.run.smithi031.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 598 kB/s | 79 kB 00:00 2023-09-07T11:00:25.327 INFO:teuthology.orchestra.run.smithi031.stdout:(2/2): ceph-fuse-18.0.0-5797.g7075f693.el9.x86_ 787 kB/s | 825 kB 00:01 2023-09-07T11:00:25.329 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-07T11:00:25.330 INFO:teuthology.orchestra.run.smithi031.stdout:Total 859 kB/s | 903 kB 00:01 2023-09-07T11:00:25.330 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-07T11:00:25.354 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-07T11:00:25.355 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-07T11:00:25.437 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-07T11:00:25.438 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-07T11:00:25.624 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-07T11:00:25.771 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-09-07T11:00:25.838 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-fuse-2:18.0.0-5797.g7075f693.el9.x86_64 2/2 2023-09-07T11:00:26.547 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-fuse-2:18.0.0-5797.g7075f693.el9.x86_64 2/2 2023-09-07T11:00:26.548 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-fuse-2:18.0.0-5797.g7075f693.el9.x86_64 1/2 2023-09-07T11:00:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-09-07T11:00:26.864 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:26.864 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-07T11:00:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse-2:18.0.0-5797.g7075f693.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-09-07T11:00:26.865 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:26.865 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T11:00:27.114 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-volume 2023-09-07T11:00:27.681 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:48 ago on Thu 07 Sep 2023 10:57:39 AM UTC. 2023-09-07T11:00:27.825 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T11:00:27.825 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T11:00:27.826 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-09-07T11:00:27.826 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T11:00:27.826 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-07T11:00:27.826 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume noarch 2:18.0.0-5797.g7075f693.el9 ceph-noarch 261 k 2023-09-07T11:00:27.827 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:27.827 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-07T11:00:27.827 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T11:00:27.827 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-09-07T11:00:27.827 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:27.828 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 261 k 2023-09-07T11:00:27.828 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 1.2 M 2023-09-07T11:00:27.828 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-07T11:00:28.549 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-volume-18.0.0-5797.g7075f693.el9.noarch.rp 362 kB/s | 261 kB 00:00 2023-09-07T11:00:28.550 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-07T11:00:28.550 INFO:teuthology.orchestra.run.smithi031.stdout:Total 362 kB/s | 261 kB 00:00 2023-09-07T11:00:28.550 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-07T11:00:28.555 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-07T11:00:28.556 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-07T11:00:28.596 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-07T11:00:28.597 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-07T11:00:28.818 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-07T11:00:28.899 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-volume-2:18.0.0-5797.g7075f693.el9.noarch 1/1 2023-09-07T11:00:28.915 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-volume-2:18.0.0-5797.g7075f693.el9.noarch 1/1 2023-09-07T11:00:28.915 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-09-07T11:00:28.915 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-09-07T11:00:28.916 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:29.927 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-volume-2:18.0.0-5797.g7075f693.el9.noarch 1/1 2023-09-07T11:00:29.927 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:29.928 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-07T11:00:29.928 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume-2:18.0.0-5797.g7075f693.el9.noarch 2023-09-07T11:00:29.928 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:29.928 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T11:00:30.239 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librados-devel 2023-09-07T11:00:30.815 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:51 ago on Thu 07 Sep 2023 10:57:39 AM UTC. 2023-09-07T11:00:30.959 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T11:00:30.959 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T11:00:30.959 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2023-09-07T11:00:30.960 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T11:00:30.960 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-07T11:00:30.960 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel x86_64 2:18.0.0-5797.g7075f693.el9 ceph 127 k 2023-09-07T11:00:30.960 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:30.960 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-07T11:00:30.961 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T11:00:30.961 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-09-07T11:00:30.961 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:30.961 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 127 k 2023-09-07T11:00:30.961 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 456 k 2023-09-07T11:00:30.962 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-07T11:00:31.167 INFO:teuthology.orchestra.run.smithi031.stdout:librados-devel-18.0.0-5797.g7075f693.el9.x86_64 611 kB/s | 127 kB 00:00 2023-09-07T11:00:31.167 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-07T11:00:31.168 INFO:teuthology.orchestra.run.smithi031.stdout:Total 607 kB/s | 127 kB 00:00 2023-09-07T11:00:31.168 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-07T11:00:31.172 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-07T11:00:31.173 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-07T11:00:31.231 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-07T11:00:31.231 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-07T11:00:31.383 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-07T11:00:31.483 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librados-devel-2:18.0.0-5797.g7075f693.el9.x86_64 1/1 2023-09-07T11:00:31.918 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados-devel-2:18.0.0-5797.g7075f693.el9.x86_64 1/1 2023-09-07T11:00:32.230 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados-devel-2:18.0.0-5797.g7075f693.el9.x86_64 1/1 2023-09-07T11:00:32.231 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:32.231 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-07T11:00:32.231 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T11:00:32.231 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:32.232 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T11:00:32.459 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install libcephfs2 2023-09-07T11:00:33.023 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:54 ago on Thu 07 Sep 2023 10:57:39 AM UTC. 2023-09-07T11:00:33.113 INFO:teuthology.orchestra.run.smithi031.stdout:Package libcephfs2-2:18.0.0-5797.g7075f693.el9.x86_64 is already installed. 2023-09-07T11:00:33.167 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T11:00:33.168 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-09-07T11:00:33.168 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T11:00:33.235 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install libcephfs-devel 2023-09-07T11:00:33.819 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:54 ago on Thu 07 Sep 2023 10:57:39 AM UTC. 2023-09-07T11:00:33.959 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T11:00:33.960 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T11:00:33.960 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2023-09-07T11:00:33.960 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T11:00:33.961 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-07T11:00:33.961 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel x86_64 2:18.0.0-5797.g7075f693.el9 ceph 31 k 2023-09-07T11:00:33.961 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:33.961 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-07T11:00:33.961 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T11:00:33.962 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-09-07T11:00:33.962 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:33.962 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 31 k 2023-09-07T11:00:33.962 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 140 k 2023-09-07T11:00:33.962 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-07T11:00:34.119 INFO:teuthology.orchestra.run.smithi031.stdout:libcephfs-devel-18.0.0-5797.g7075f693.el9.x86_6 198 kB/s | 31 kB 00:00 2023-09-07T11:00:34.119 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-07T11:00:34.120 INFO:teuthology.orchestra.run.smithi031.stdout:Total 196 kB/s | 31 kB 00:00 2023-09-07T11:00:34.120 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-07T11:00:34.123 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-07T11:00:34.123 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-07T11:00:34.149 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-07T11:00:34.149 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-07T11:00:34.246 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-07T11:00:34.352 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs-devel-2:18.0.0-5797.g7075f693.el9.x86_64 1/1 2023-09-07T11:00:34.761 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-5797.g7075f693.el9.x86_64 1/1 2023-09-07T11:00:35.036 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs-devel-2:18.0.0-5797.g7075f693.el9.x86_64 1/1 2023-09-07T11:00:35.036 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:35.036 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-07T11:00:35.036 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T11:00:35.037 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:35.037 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T11:00:35.204 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librados2 2023-09-07T11:00:35.772 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:56 ago on Thu 07 Sep 2023 10:57:39 AM UTC. 2023-09-07T11:00:35.861 INFO:teuthology.orchestra.run.smithi031.stdout:Package librados2-2:18.0.0-5797.g7075f693.el9.x86_64 is already installed. 2023-09-07T11:00:35.912 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T11:00:35.913 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-09-07T11:00:35.913 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T11:00:35.973 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librbd1 2023-09-07T11:00:36.537 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:57 ago on Thu 07 Sep 2023 10:57:39 AM UTC. 2023-09-07T11:00:36.626 INFO:teuthology.orchestra.run.smithi031.stdout:Package librbd1-2:18.0.0-5797.g7075f693.el9.x86_64 is already installed. 2023-09-07T11:00:36.678 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T11:00:36.684 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-09-07T11:00:36.684 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T11:00:36.739 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rados 2023-09-07T11:00:37.299 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:58 ago on Thu 07 Sep 2023 10:57:39 AM UTC. 2023-09-07T11:00:37.388 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rados-2:18.0.0-5797.g7075f693.el9.x86_64 is already installed. 2023-09-07T11:00:37.440 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T11:00:37.441 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-09-07T11:00:37.441 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T11:00:37.497 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rgw 2023-09-07T11:00:38.056 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:59 ago on Thu 07 Sep 2023 10:57:39 AM UTC. 2023-09-07T11:00:38.149 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rgw-2:18.0.0-5797.g7075f693.el9.x86_64 is already installed. 2023-09-07T11:00:38.202 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T11:00:38.204 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-09-07T11:00:38.204 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T11:00:38.263 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-cephfs 2023-09-07T11:00:38.833 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:59 ago on Thu 07 Sep 2023 10:57:39 AM UTC. 2023-09-07T11:00:38.922 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-cephfs-2:18.0.0-5797.g7075f693.el9.x86_64 is already installed. 2023-09-07T11:00:38.976 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T11:00:38.977 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-09-07T11:00:38.977 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T11:00:39.039 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rbd 2023-09-07T11:00:39.596 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:00 ago on Thu 07 Sep 2023 10:57:39 AM UTC. 2023-09-07T11:00:39.687 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rbd-2:18.0.0-5797.g7075f693.el9.x86_64 is already installed. 2023-09-07T11:00:39.742 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T11:00:39.742 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-09-07T11:00:39.742 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T11:00:39.797 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-fuse 2023-09-07T11:00:40.365 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:01 ago on Thu 07 Sep 2023 10:57:39 AM UTC. 2023-09-07T11:00:40.507 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T11:00:40.507 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T11:00:40.508 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2023-09-07T11:00:40.508 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T11:00:40.508 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-07T11:00:40.508 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse x86_64 2:18.0.0-5797.g7075f693.el9 ceph 87 k 2023-09-07T11:00:40.508 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:40.509 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-07T11:00:40.509 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T11:00:40.509 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-09-07T11:00:40.509 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:40.509 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 87 k 2023-09-07T11:00:40.510 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 252 k 2023-09-07T11:00:40.510 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-07T11:00:40.717 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-fuse-18.0.0-5797.g7075f693.el9.x86_64.rpm 423 kB/s | 87 kB 00:00 2023-09-07T11:00:40.718 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-07T11:00:40.718 INFO:teuthology.orchestra.run.smithi031.stdout:Total 420 kB/s | 87 kB 00:00 2023-09-07T11:00:40.718 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-07T11:00:40.721 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-07T11:00:40.722 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-07T11:00:40.779 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-07T11:00:40.780 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-07T11:00:40.932 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-07T11:00:41.030 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-fuse-2:18.0.0-5797.g7075f693.el9.x86_64 1/1 2023-09-07T11:00:41.427 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-fuse-2:18.0.0-5797.g7075f693.el9.x86_64 1/1 2023-09-07T11:00:41.742 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-fuse-2:18.0.0-5797.g7075f693.el9.x86_64 1/1 2023-09-07T11:00:41.743 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:41.743 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-07T11:00:41.743 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T11:00:41.743 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:41.743 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T11:00:41.982 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-mirror 2023-09-07T11:00:42.541 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:03 ago on Thu 07 Sep 2023 10:57:39 AM UTC. 2023-09-07T11:00:42.686 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T11:00:42.686 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T11:00:42.687 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2023-09-07T11:00:42.687 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T11:00:42.687 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-07T11:00:42.687 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror x86_64 2:18.0.0-5797.g7075f693.el9 ceph 3.0 M 2023-09-07T11:00:42.687 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:42.688 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-07T11:00:42.688 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T11:00:42.688 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-09-07T11:00:42.688 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:42.688 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 3.0 M 2023-09-07T11:00:42.689 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 14 M 2023-09-07T11:00:42.689 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-07T11:00:43.882 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-mirror-18.0.0-5797.g7075f693.el9.x86_64.rpm 2.5 MB/s | 3.0 MB 00:01 2023-09-07T11:00:43.883 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-07T11:00:43.884 INFO:teuthology.orchestra.run.smithi031.stdout:Total 2.5 MB/s | 3.0 MB 00:01 2023-09-07T11:00:43.884 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-07T11:00:43.896 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-07T11:00:43.896 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-07T11:00:43.972 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-07T11:00:43.973 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-07T11:00:44.418 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-07T11:00:44.483 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-mirror-2:18.0.0-5797.g7075f693.el9.x86_64 1/1 2023-09-07T11:00:44.507 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5797.g7075f693.el9.x86_64 1/1 2023-09-07T11:00:44.507 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-09-07T11:00:44.508 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-09-07T11:00:44.508 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-09-07T11:00:44.508 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-09-07T11:00:44.508 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:45.691 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-mirror-2:18.0.0-5797.g7075f693.el9.x86_64 1/1 2023-09-07T11:00:45.691 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:45.692 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-07T11:00:45.692 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T11:00:45.692 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:45.692 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T11:00:45.955 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-nbd 2023-09-07T11:00:46.522 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:07 ago on Thu 07 Sep 2023 10:57:39 AM UTC. 2023-09-07T11:00:46.663 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-09-07T11:00:46.664 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T11:00:46.665 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2023-09-07T11:00:46.665 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T11:00:46.665 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-09-07T11:00:46.665 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd x86_64 2:18.0.0-5797.g7075f693.el9 ceph 168 k 2023-09-07T11:00:46.665 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:46.666 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-09-07T11:00:46.666 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-09-07T11:00:46.666 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-09-07T11:00:46.666 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:46.667 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 168 k 2023-09-07T11:00:46.667 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 532 k 2023-09-07T11:00:46.667 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-09-07T11:00:46.996 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-nbd-18.0.0-5797.g7075f693.el9.x86_64.rpm 512 kB/s | 168 kB 00:00 2023-09-07T11:00:46.997 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-09-07T11:00:46.997 INFO:teuthology.orchestra.run.smithi031.stdout:Total 510 kB/s | 168 kB 00:00 2023-09-07T11:00:46.997 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-09-07T11:00:47.004 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-09-07T11:00:47.004 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-09-07T11:00:47.062 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-09-07T11:00:47.062 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-09-07T11:00:47.220 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-09-07T11:00:47.329 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-nbd-2:18.0.0-5797.g7075f693.el9.x86_64 1/1 2023-09-07T11:00:47.744 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-nbd-2:18.0.0-5797.g7075f693.el9.x86_64 1/1 2023-09-07T11:00:48.041 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-nbd-2:18.0.0-5797.g7075f693.el9.x86_64 1/1 2023-09-07T11:00:48.041 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:48.042 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-09-07T11:00:48.042 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd-2:18.0.0-5797.g7075f693.el9.x86_64 2023-09-07T11:00:48.042 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:48.042 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-09-07T11:00:48.265 DEBUG:teuthology.parallel:result is None 2023-09-07T11:00:48.266 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=7075f69375d3b106d3a94dda948a86d616371099 2023-09-07T11:00:48.495 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-07T11:00:48.517 INFO:teuthology.orchestra.run.smithi031.stdout:18.0.0-5797.g7075f693.el9 2023-09-07T11:00:48.517 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5797.g7075f693.el9 2023-09-07T11:00:48.518 INFO:teuthology.task.install:The correct ceph version 18.0.0-5797.g7075f693 is installed. 2023-09-07T11:00:48.518 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-07T11:00:48.519 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-07T11:00:48.519 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-07T11:00:48.589 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-07T11:00:48.590 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-07T11:00:48.590 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/daemon-helper 2023-09-07T11:00:48.666 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-07T11:00:48.738 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-07T11:00:48.738 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-07T11:00:48.739 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-07T11:00:48.810 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-07T11:00:48.886 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-07T11:00:48.887 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-07T11:00:48.887 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/stdin-killer 2023-09-07T11:00:48.956 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-07T11:00:49.030 INFO:teuthology.run_tasks:Running task ceph... 2023-09-07T11:00:49.172 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-07T11:00:49.173 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 777 /var/log/ceph 2023-09-07T11:00:49.209 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-07T11:00:49.210 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-07T11:00:49.279 INFO:tasks.ceph:Creating extra log directories... 2023-09-07T11:00:49.280 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-07T11:00:49.365 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-09-07T11:00:49.366 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-07T11:00:49.367 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2023-09-07_10:45:43-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-ref-crimson-only-distro-crimson-smithi/7390804', 'branch': 'wip-matanb-crimson-do_osd_ops_execute-ref-crimson-only', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read}', 'email': None, 'first_in_suite': False, 'job_id': '7390804', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2023-09-07_10:45:43-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-ref-crimson-only-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-do_osd_ops_execute-ref-crimson-only'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '7075f69375d3b106d3a94dda948a86d616371099'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '7075f69375d3b106d3a94dda948a86d616371099'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'f4c5809eee86803e43be24d3362e60639f88c948'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 1657, 'sha1': '7075f69375d3b106d3a94dda948a86d616371099', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_f4c5809eee86803e43be24d3362e60639f88c948/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': 'f4c5809eee86803e43be24d3362e60639f88c948', 'targets': {'smithi031.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGaX5e7rZpui2YElhU102jZW5H2XNqBZ4e0SP9T451dXzh+rj/ICxhb9YL6Yl+7gbPr8wbmGmold07FAy9yrfRk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '7075f69375d3b106d3a94dda948a86d616371099', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-07_10:45:43', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1394'} 2023-09-07T11:00:49.369 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-07T11:00:49.411 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/run/ceph 2023-09-07T11:00:49.482 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-07T11:00:49.483 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout 2023-09-07T11:00:49.542 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-07T11:00:49.543 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1 2023-09-07T11:00:49.597 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-07T11:00:49.598 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-07T11:00:49.598 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 663 Links: 1 2023-09-07T11:00:49.598 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-07T11:00:49.598 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-09-07T11:00:49.599 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-09-07 11:00:44.779460105 +0000 2023-09-07T11:00:49.599 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-09-07 10:56:40.776344443 +0000 2023-09-07T11:00:49.599 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-09-07 10:56:40.776344443 +0000 2023-09-07T11:00:49.599 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2023-09-07 10:56:40.776344443 +0000 2023-09-07T11:00:49.600 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-07T11:00:49.667 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-09-07T11:00:49.667 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-09-07T11:00:49.667 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000198125 s, 2.6 MB/s 2023-09-07T11:00:49.668 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-07T11:00:49.724 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2 2023-09-07T11:00:49.785 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-07T11:00:49.785 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-07T11:00:49.786 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 684 Links: 1 2023-09-07T11:00:49.786 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-07T11:00:49.786 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-09-07T11:00:49.787 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-09-07 11:00:44.779460105 +0000 2023-09-07T11:00:49.787 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-09-07 10:56:41.284336551 +0000 2023-09-07T11:00:49.787 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-09-07 10:56:41.284336551 +0000 2023-09-07T11:00:49.787 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2023-09-07 10:56:41.284336551 +0000 2023-09-07T11:00:49.788 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-07T11:00:49.856 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-09-07T11:00:49.857 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-09-07T11:00:49.857 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000178602 s, 2.9 MB/s 2023-09-07T11:00:49.857 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-07T11:00:49.912 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3 2023-09-07T11:00:49.967 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-07T11:00:49.968 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-07T11:00:49.968 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-09-07T11:00:49.968 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-07T11:00:49.969 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-09-07T11:00:49.969 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-09-07 11:00:44.780460089 +0000 2023-09-07T11:00:49.969 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-09-07 10:56:41.805328458 +0000 2023-09-07T11:00:49.969 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-09-07 10:56:41.805328458 +0000 2023-09-07T11:00:49.970 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2023-09-07 10:56:41.805328458 +0000 2023-09-07T11:00:49.970 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-07T11:00:50.036 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-09-07T11:00:50.036 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-09-07T11:00:50.037 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000186538 s, 2.7 MB/s 2023-09-07T11:00:50.038 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-07T11:00:50.098 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4 2023-09-07T11:00:50.153 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-07T11:00:50.154 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-07T11:00:50.154 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 722 Links: 1 2023-09-07T11:00:50.154 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-07T11:00:50.154 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-09-07T11:00:50.155 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-09-07 11:00:44.780460089 +0000 2023-09-07T11:00:50.155 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-09-07 10:56:42.314320550 +0000 2023-09-07T11:00:50.155 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-09-07 10:56:42.314320550 +0000 2023-09-07T11:00:50.155 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2023-09-07 10:56:42.314320550 +0000 2023-09-07T11:00:50.156 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-07T11:00:50.222 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-09-07T11:00:50.222 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-09-07T11:00:50.223 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000250973 s, 2.0 MB/s 2023-09-07T11:00:50.224 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-07T11:00:50.280 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-07T11:00:50.280 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-09-07T11:00:50.281 INFO:tasks.ceph:Generating config... 2023-09-07T11:00:50.282 INFO:tasks.ceph:[osd] debug monc = 20 2023-09-07T11:00:50.283 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-09-07T11:00:50.283 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2023-09-07T11:00:50.283 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-09-07T11:00:50.283 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-09-07T11:00:50.284 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2023-09-07T11:00:50.284 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2023-09-07T11:00:50.292 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-09-07T11:00:50.294 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-09-07T11:00:50.295 INFO:tasks.ceph:[osd] debug filestore = 0/0 2023-09-07T11:00:50.295 INFO:tasks.ceph:[osd] debug journal = 0/0 2023-09-07T11:00:50.303 INFO:tasks.ceph:[osd] debug ms = 0/0 2023-09-07T11:00:50.304 INFO:tasks.ceph:[osd] debug osd = 0/0 2023-09-07T11:00:50.305 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-09-07T11:00:50.305 INFO:tasks.ceph:[osd] err to stderr = True 2023-09-07T11:00:50.305 INFO:tasks.ceph:[osd] log flush on exit = True 2023-09-07T11:00:50.306 INFO:tasks.ceph:[osd] log to file = False 2023-09-07T11:00:50.312 INFO:tasks.ceph:[osd] log to stderr = True 2023-09-07T11:00:50.312 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-09-07T11:00:50.312 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-09-07T11:00:50.313 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-09-07T11:00:50.313 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-09-07T11:00:50.314 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-09-07T11:00:50.315 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-09-07T11:00:50.315 INFO:tasks.ceph:[global] auth client required = none 2023-09-07T11:00:50.315 INFO:tasks.ceph:[global] auth cluster required = none 2023-09-07T11:00:50.315 INFO:tasks.ceph:[global] auth service required = none 2023-09-07T11:00:50.316 INFO:tasks.ceph:[global] auth supported = none 2023-09-07T11:00:50.316 INFO:tasks.ceph:[global] debug asok = 0/0 2023-09-07T11:00:50.316 INFO:tasks.ceph:[global] debug auth = 0/0 2023-09-07T11:00:50.316 INFO:tasks.ceph:[global] debug bdev = 0/0 2023-09-07T11:00:50.317 INFO:tasks.ceph:[global] debug bluefs = 0/0 2023-09-07T11:00:50.317 INFO:tasks.ceph:[global] debug bluestore = 0/0 2023-09-07T11:00:50.318 INFO:tasks.ceph:[global] debug buffer = 0/0 2023-09-07T11:00:50.318 INFO:tasks.ceph:[global] debug civetweb = 0/0 2023-09-07T11:00:50.318 INFO:tasks.ceph:[global] debug client = 0/0 2023-09-07T11:00:50.324 INFO:tasks.ceph:[global] debug compressor = 0/0 2023-09-07T11:00:50.325 INFO:tasks.ceph:[global] debug context = 0/0 2023-09-07T11:00:50.325 INFO:tasks.ceph:[global] debug crush = 0/0 2023-09-07T11:00:50.325 INFO:tasks.ceph:[global] debug crypto = 0/0 2023-09-07T11:00:50.326 INFO:tasks.ceph:[global] debug dpdk = 0/0 2023-09-07T11:00:50.326 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2023-09-07T11:00:50.326 INFO:tasks.ceph:[global] debug filer = 0/0 2023-09-07T11:00:50.326 INFO:tasks.ceph:[global] debug filestore = 0/0 2023-09-07T11:00:50.331 INFO:tasks.ceph:[global] debug finisher = 0/0 2023-09-07T11:00:50.332 INFO:tasks.ceph:[global] debug fuse = 0/0 2023-09-07T11:00:50.332 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2023-09-07T11:00:50.332 INFO:tasks.ceph:[global] debug javaclient = 0/0 2023-09-07T11:00:50.333 INFO:tasks.ceph:[global] debug journal = 0/0 2023-09-07T11:00:50.334 INFO:tasks.ceph:[global] debug journaler = 0/0 2023-09-07T11:00:50.336 INFO:tasks.ceph:[global] debug kstore = 0/0 2023-09-07T11:00:50.336 INFO:tasks.ceph:[global] debug leveldb = 0/0 2023-09-07T11:00:50.336 INFO:tasks.ceph:[global] debug lockdep = 0/0 2023-09-07T11:00:50.336 INFO:tasks.ceph:[global] debug mds = 0/0 2023-09-07T11:00:50.337 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2023-09-07T11:00:50.337 INFO:tasks.ceph:[global] debug mds locker = 0/0 2023-09-07T11:00:50.337 INFO:tasks.ceph:[global] debug mds log = 0/0 2023-09-07T11:00:50.337 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2023-09-07T11:00:50.338 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2023-09-07T11:00:50.338 INFO:tasks.ceph:[global] debug memdb = 0/0 2023-09-07T11:00:50.338 INFO:tasks.ceph:[global] debug mgr = 0/0 2023-09-07T11:00:50.338 INFO:tasks.ceph:[global] debug mgrc = 0/0 2023-09-07T11:00:50.339 INFO:tasks.ceph:[global] debug mon = 0/0 2023-09-07T11:00:50.339 INFO:tasks.ceph:[global] debug monc = 0/0 2023-09-07T11:00:50.339 INFO:tasks.ceph:[global] debug ms = 0/0 2023-09-07T11:00:50.339 INFO:tasks.ceph:[global] debug objclass = 0/0 2023-09-07T11:00:50.340 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2023-09-07T11:00:50.340 INFO:tasks.ceph:[global] debug objecter = 0/0 2023-09-07T11:00:50.340 INFO:tasks.ceph:[global] debug optracker = 0/0 2023-09-07T11:00:50.341 INFO:tasks.ceph:[global] debug osd = 0/0 2023-09-07T11:00:50.341 INFO:tasks.ceph:[global] debug paxos = 0/0 2023-09-07T11:00:50.341 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2023-09-07T11:00:50.342 INFO:tasks.ceph:[global] debug rados = 0/0 2023-09-07T11:00:50.342 INFO:tasks.ceph:[global] debug rbd = 0/0 2023-09-07T11:00:50.342 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2023-09-07T11:00:50.343 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2023-09-07T11:00:50.343 INFO:tasks.ceph:[global] debug refs = 0/0 2023-09-07T11:00:50.343 INFO:tasks.ceph:[global] debug rgw = 0/0 2023-09-07T11:00:50.344 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2023-09-07T11:00:50.345 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2023-09-07T11:00:50.345 INFO:tasks.ceph:[global] debug striper = 0/0 2023-09-07T11:00:50.346 INFO:tasks.ceph:[global] debug throttle = 0/0 2023-09-07T11:00:50.346 INFO:tasks.ceph:[global] debug timer = 0/0 2023-09-07T11:00:50.347 INFO:tasks.ceph:[global] debug tp = 0/0 2023-09-07T11:00:50.347 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2023-09-07T11:00:50.347 INFO:tasks.ceph:[global] ms client mode = crc 2023-09-07T11:00:50.347 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-09-07T11:00:50.348 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-09-07T11:00:50.348 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-09-07T11:00:50.348 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-09-07T11:00:50.348 INFO:tasks.ceph:[global] ms service mode = crc 2023-09-07T11:00:50.349 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-07T11:00:50.349 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-07T11:00:50.349 INFO:tasks.ceph:[mon] debug mon = 0/0 2023-09-07T11:00:50.349 INFO:tasks.ceph:[mon] debug ms = 0/0 2023-09-07T11:00:50.350 INFO:tasks.ceph:[mon] debug paxos = 0/0 2023-09-07T11:00:50.350 INFO:tasks.ceph:[mon] osd pool default crimson = True 2023-09-07T11:00:50.350 INFO:tasks.ceph:Setting up mon.a... 2023-09-07T11:00:50.351 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-09-07T11:00:50.413 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/ceph.keyring 2023-09-07T11:00:50.416 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-09-07T11:00:50.476 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-07T11:00:50.507 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.31')] 2023-09-07T11:00:50.508 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.31', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-09-07T11:00:50.509 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-07T11:00:50.510 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-07T11:00:50.563 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.31 --print /home/ubuntu/cephtest/ceph.monmap 2023-09-07T11:00:50.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:00:50.648+0000 7f3fea829e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:00:50.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:00:50.648+0000 7f3fea829e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:00:50.649 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:00:50.648+0000 7f3fea829e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:00:50.651 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-09-07T11:00:50.651 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: generated fsid e8f5877e-ee15-4b77-b5d2-4d38002bcd0c 2023-09-07T11:00:50.651 INFO:teuthology.orchestra.run.smithi031.stdout:setting min_mon_release = pacific 2023-09-07T11:00:50.651 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 0 2023-09-07T11:00:50.651 INFO:teuthology.orchestra.run.smithi031.stdout:fsid e8f5877e-ee15-4b77-b5d2-4d38002bcd0c 2023-09-07T11:00:50.652 INFO:teuthology.orchestra.run.smithi031.stdout:last_changed 2023-09-07T11:00:50.649599+0000 2023-09-07T11:00:50.652 INFO:teuthology.orchestra.run.smithi031.stdout:created 2023-09-07T11:00:50.649599+0000 2023-09-07T11:00:50.652 INFO:teuthology.orchestra.run.smithi031.stdout:min_mon_release 16 (pacific) 2023-09-07T11:00:50.659 INFO:teuthology.orchestra.run.smithi031.stdout:election_strategy: 1 2023-09-07T11:00:50.659 INFO:teuthology.orchestra.run.smithi031.stdout:0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2023-09-07T11:00:50.659 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-09-07T11:00:50.660 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-07T11:00:50.675 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e8f5877e-ee15-4b77-b5d2-4d38002bcd0c... 2023-09-07T11:00:50.676 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-07T11:00:50.775 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2023-09-07T11:00:50.776 INFO:teuthology.orchestra.run.smithi031.stdout: chdir = "" 2023-09-07T11:00:50.776 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-07T11:00:50.776 INFO:teuthology.orchestra.run.smithi031.stdout: auth supported = none 2023-09-07T11:00:50.776 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.777 INFO:teuthology.orchestra.run.smithi031.stdout: filestore xattr use omap = true 2023-09-07T11:00:50.777 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.777 INFO:teuthology.orchestra.run.smithi031.stdout: mon clock drift allowed = 1.000 2023-09-07T11:00:50.778 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.778 INFO:teuthology.orchestra.run.smithi031.stdout: osd crush chooseleaf type = 0 2023-09-07T11:00:50.778 INFO:teuthology.orchestra.run.smithi031.stdout: auth debug = true 2023-09-07T11:00:50.778 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.778 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on old message = true 2023-09-07T11:00:50.779 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on bug = true 2023-09-07T11:00:50.779 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.779 INFO:teuthology.orchestra.run.smithi031.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-07T11:00:50.779 INFO:teuthology.orchestra.run.smithi031.stdout: mon pg warn max object skew = 0 2023-09-07T11:00:50.780 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.780 INFO:teuthology.orchestra.run.smithi031.stdout: # disable pg_autoscaler by default for new pools 2023-09-07T11:00:50.781 INFO:teuthology.orchestra.run.smithi031.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-07T11:00:50.781 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.781 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default size = 2 2023-09-07T11:00:50.782 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.782 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow primary affinity = true 2023-09-07T11:00:50.782 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow pg remap = true 2023-09-07T11:00:50.782 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on legacy crush tunables = false 2023-09-07T11:00:50.783 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on crush straw calc version zero = false 2023-09-07T11:00:50.783 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on no sortbitwise = false 2023-09-07T11:00:50.783 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on osd down out interval zero = false 2023-09-07T11:00:50.783 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on too few osds = false 2023-09-07T11:00:50.784 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-07T11:00:50.784 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-07T11:00:50.784 INFO:teuthology.orchestra.run.smithi031.stdout: mon_allow_pool_size_one = true 2023-09-07T11:00:50.784 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.784 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-07T11:00:50.785 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.785 INFO:teuthology.orchestra.run.smithi031.stdout: osd default data pool replay window = 5 2023-09-07T11:00:50.785 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.786 INFO:teuthology.orchestra.run.smithi031.stdout: mon allow pool delete = true 2023-09-07T11:00:50.786 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.786 INFO:teuthology.orchestra.run.smithi031.stdout: mon cluster log file level = debug 2023-09-07T11:00:50.786 INFO:teuthology.orchestra.run.smithi031.stdout: debug asserts on shutdown = true 2023-09-07T11:00:50.787 INFO:teuthology.orchestra.run.smithi031.stdout: mon health detail to clog = false 2023-09-07T11:00:50.787 INFO:teuthology.orchestra.run.smithi031.stdout: mon host = 172.21.15.31 2023-09-07T11:00:50.787 INFO:teuthology.orchestra.run.smithi031.stdout: auth client required = none 2023-09-07T11:00:50.787 INFO:teuthology.orchestra.run.smithi031.stdout: auth cluster required = none 2023-09-07T11:00:50.788 INFO:teuthology.orchestra.run.smithi031.stdout: auth service required = none 2023-09-07T11:00:50.788 INFO:teuthology.orchestra.run.smithi031.stdout: debug asok = 0/0 2023-09-07T11:00:50.788 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 0/0 2023-09-07T11:00:50.788 INFO:teuthology.orchestra.run.smithi031.stdout: debug bdev = 0/0 2023-09-07T11:00:50.788 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluefs = 0/0 2023-09-07T11:00:50.789 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluestore = 0/0 2023-09-07T11:00:50.789 INFO:teuthology.orchestra.run.smithi031.stdout: debug buffer = 0/0 2023-09-07T11:00:50.789 INFO:teuthology.orchestra.run.smithi031.stdout: debug civetweb = 0/0 2023-09-07T11:00:50.789 INFO:teuthology.orchestra.run.smithi031.stdout: debug client = 0/0 2023-09-07T11:00:50.789 INFO:teuthology.orchestra.run.smithi031.stdout: debug compressor = 0/0 2023-09-07T11:00:50.790 INFO:teuthology.orchestra.run.smithi031.stdout: debug context = 0/0 2023-09-07T11:00:50.790 INFO:teuthology.orchestra.run.smithi031.stdout: debug crush = 0/0 2023-09-07T11:00:50.790 INFO:teuthology.orchestra.run.smithi031.stdout: debug crypto = 0/0 2023-09-07T11:00:50.791 INFO:teuthology.orchestra.run.smithi031.stdout: debug dpdk = 0/0 2023-09-07T11:00:50.791 INFO:teuthology.orchestra.run.smithi031.stdout: debug eventtrace = 0/0 2023-09-07T11:00:50.791 INFO:teuthology.orchestra.run.smithi031.stdout: debug filer = 0/0 2023-09-07T11:00:50.792 INFO:teuthology.orchestra.run.smithi031.stdout: debug filestore = 0/0 2023-09-07T11:00:50.792 INFO:teuthology.orchestra.run.smithi031.stdout: debug finisher = 0/0 2023-09-07T11:00:50.792 INFO:teuthology.orchestra.run.smithi031.stdout: debug fuse = 0/0 2023-09-07T11:00:50.793 INFO:teuthology.orchestra.run.smithi031.stdout: debug heartbeatmap = 0/0 2023-09-07T11:00:50.793 INFO:teuthology.orchestra.run.smithi031.stdout: debug javaclient = 0/0 2023-09-07T11:00:50.793 INFO:teuthology.orchestra.run.smithi031.stdout: debug journal = 0/0 2023-09-07T11:00:50.793 INFO:teuthology.orchestra.run.smithi031.stdout: debug journaler = 0/0 2023-09-07T11:00:50.798 INFO:teuthology.orchestra.run.smithi031.stdout: debug kstore = 0/0 2023-09-07T11:00:50.802 INFO:teuthology.orchestra.run.smithi031.stdout: debug leveldb = 0/0 2023-09-07T11:00:50.802 INFO:teuthology.orchestra.run.smithi031.stdout: debug lockdep = 0/0 2023-09-07T11:00:50.802 INFO:teuthology.orchestra.run.smithi031.stdout: debug mds = 0/0 2023-09-07T11:00:50.802 INFO:teuthology.orchestra.run.smithi031.stdout: debug mds balancer = 0/0 2023-09-07T11:00:50.802 INFO:teuthology.orchestra.run.smithi031.stdout: debug mds locker = 0/0 2023-09-07T11:00:50.803 INFO:teuthology.orchestra.run.smithi031.stdout: debug mds log = 0/0 2023-09-07T11:00:50.803 INFO:teuthology.orchestra.run.smithi031.stdout: debug mds log expire = 0/0 2023-09-07T11:00:50.803 INFO:teuthology.orchestra.run.smithi031.stdout: debug mds migrator = 0/0 2023-09-07T11:00:50.803 INFO:teuthology.orchestra.run.smithi031.stdout: debug memdb = 0/0 2023-09-07T11:00:50.803 INFO:teuthology.orchestra.run.smithi031.stdout: debug mgr = 0/0 2023-09-07T11:00:50.804 INFO:teuthology.orchestra.run.smithi031.stdout: debug mgrc = 0/0 2023-09-07T11:00:50.804 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 0/0 2023-09-07T11:00:50.804 INFO:teuthology.orchestra.run.smithi031.stdout: debug monc = 0/0 2023-09-07T11:00:50.804 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 0/0 2023-09-07T11:00:50.804 INFO:teuthology.orchestra.run.smithi031.stdout: debug objclass = 0/0 2023-09-07T11:00:50.805 INFO:teuthology.orchestra.run.smithi031.stdout: debug objectcacher = 0/0 2023-09-07T11:00:50.805 INFO:teuthology.orchestra.run.smithi031.stdout: debug objecter = 0/0 2023-09-07T11:00:50.805 INFO:teuthology.orchestra.run.smithi031.stdout: debug optracker = 0/0 2023-09-07T11:00:50.805 INFO:teuthology.orchestra.run.smithi031.stdout: debug osd = 0/0 2023-09-07T11:00:50.805 INFO:teuthology.orchestra.run.smithi031.stdout: debug paxos = 0/0 2023-09-07T11:00:50.805 INFO:teuthology.orchestra.run.smithi031.stdout: debug perfcounter = 0/0 2023-09-07T11:00:50.806 INFO:teuthology.orchestra.run.smithi031.stdout: debug rados = 0/0 2023-09-07T11:00:50.806 INFO:teuthology.orchestra.run.smithi031.stdout: debug rbd = 0/0 2023-09-07T11:00:50.806 INFO:teuthology.orchestra.run.smithi031.stdout: debug rbd mirror = 0/0 2023-09-07T11:00:50.806 INFO:teuthology.orchestra.run.smithi031.stdout: debug rbd replay = 0/0 2023-09-07T11:00:50.806 INFO:teuthology.orchestra.run.smithi031.stdout: debug refs = 0/0 2023-09-07T11:00:50.807 INFO:teuthology.orchestra.run.smithi031.stdout: debug rgw = 0/0 2023-09-07T11:00:50.807 INFO:teuthology.orchestra.run.smithi031.stdout: debug rgw sync = 0/0 2023-09-07T11:00:50.825 INFO:teuthology.orchestra.run.smithi031.stdout: debug rocksdb = 0/0 2023-09-07T11:00:50.825 INFO:teuthology.orchestra.run.smithi031.stdout: debug striper = 0/0 2023-09-07T11:00:50.825 INFO:teuthology.orchestra.run.smithi031.stdout: debug throttle = 0/0 2023-09-07T11:00:50.825 INFO:teuthology.orchestra.run.smithi031.stdout: debug timer = 0/0 2023-09-07T11:00:50.826 INFO:teuthology.orchestra.run.smithi031.stdout: debug tp = 0/0 2023-09-07T11:00:50.826 INFO:teuthology.orchestra.run.smithi031.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-09-07T11:00:50.826 INFO:teuthology.orchestra.run.smithi031.stdout: ms client mode = crc 2023-09-07T11:00:50.826 INFO:teuthology.orchestra.run.smithi031.stdout: ms cluster mode = crc 2023-09-07T11:00:50.826 INFO:teuthology.orchestra.run.smithi031.stdout: ms mon client mode = crc 2023-09-07T11:00:50.833 INFO:teuthology.orchestra.run.smithi031.stdout: ms mon cluster mode = crc 2023-09-07T11:00:50.834 INFO:teuthology.orchestra.run.smithi031.stdout: ms mon service mode = crc 2023-09-07T11:00:50.834 INFO:teuthology.orchestra.run.smithi031.stdout: ms service mode = crc 2023-09-07T11:00:50.834 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = e8f5877e-ee15-4b77-b5d2-4d38002bcd0c 2023-09-07T11:00:50.834 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.834 INFO:teuthology.orchestra.run.smithi031.stdout:[osd] 2023-09-07T11:00:50.835 INFO:teuthology.orchestra.run.smithi031.stdout: osd journal size = 100 2023-09-07T11:00:50.835 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.835 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub load threshold = 5.0 2023-09-07T11:00:50.835 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub max interval = 600 2023-09-07T11:00:50.836 INFO:teuthology.orchestra.run.smithi031.stdout: osd mclock profile = high_recovery_ops 2023-09-07T11:00:50.836 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.836 INFO:teuthology.orchestra.run.smithi031.stdout: osd recover clone overlap = true 2023-09-07T11:00:50.837 INFO:teuthology.orchestra.run.smithi031.stdout: osd recovery max chunk = 1048576 2023-09-07T11:00:50.837 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.838 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug shutdown = true 2023-09-07T11:00:50.838 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug op order = true 2023-09-07T11:00:50.838 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug verify stray on activate = true 2023-09-07T11:00:50.839 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.839 INFO:teuthology.orchestra.run.smithi031.stdout: osd open classes on start = true 2023-09-07T11:00:50.839 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug pg log writeout = true 2023-09-07T11:00:50.858 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.858 INFO:teuthology.orchestra.run.smithi031.stdout: osd deep scrub update digest min age = 30 2023-09-07T11:00:50.859 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.859 INFO:teuthology.orchestra.run.smithi031.stdout: osd map max advance = 10 2023-09-07T11:00:50.859 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.859 INFO:teuthology.orchestra.run.smithi031.stdout: journal zero on create = true 2023-09-07T11:00:50.859 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.860 INFO:teuthology.orchestra.run.smithi031.stdout: filestore ondisk finisher threads = 3 2023-09-07T11:00:50.861 INFO:teuthology.orchestra.run.smithi031.stdout: filestore apply finisher threads = 3 2023-09-07T11:00:50.861 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.861 INFO:teuthology.orchestra.run.smithi031.stdout: bdev debug aio = true 2023-09-07T11:00:50.861 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug misdirected ops = true 2023-09-07T11:00:50.862 INFO:teuthology.orchestra.run.smithi031.stdout: debug monc = 20 2023-09-07T11:00:50.862 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore block size = 96636764160 2023-09-07T11:00:50.862 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore compression algorithm = snappy 2023-09-07T11:00:50.862 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore compression mode = aggressive 2023-09-07T11:00:50.863 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore fsck on mount = True 2023-09-07T11:00:50.863 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore rocksdb cf = False 2023-09-07T11:00:50.863 INFO:teuthology.orchestra.run.smithi031.stdout: crimson seastar smp = 3 2023-09-07T11:00:50.863 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluefs = 20 2023-09-07T11:00:50.863 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluestore = 20 2023-09-07T11:00:50.863 INFO:teuthology.orchestra.run.smithi031.stdout: debug filestore = 0/0 2023-09-07T11:00:50.864 INFO:teuthology.orchestra.run.smithi031.stdout: debug journal = 0/0 2023-09-07T11:00:50.864 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 0/0 2023-09-07T11:00:50.864 INFO:teuthology.orchestra.run.smithi031.stdout: debug osd = 0/0 2023-09-07T11:00:50.864 INFO:teuthology.orchestra.run.smithi031.stdout: debug rocksdb = 10 2023-09-07T11:00:50.864 INFO:teuthology.orchestra.run.smithi031.stdout: err to stderr = True 2023-09-07T11:00:50.865 INFO:teuthology.orchestra.run.smithi031.stdout: log flush on exit = True 2023-09-07T11:00:50.865 INFO:teuthology.orchestra.run.smithi031.stdout: log to file = False 2023-09-07T11:00:50.865 INFO:teuthology.orchestra.run.smithi031.stdout: log to stderr = True 2023-09-07T11:00:50.865 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd backfillfull_ratio = 0.85 2023-09-07T11:00:50.865 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd full ratio = 0.9 2023-09-07T11:00:50.866 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd nearfull ratio = 0.8 2023-09-07T11:00:50.866 INFO:teuthology.orchestra.run.smithi031.stdout: osd failsafe full ratio = 0.95 2023-09-07T11:00:50.866 INFO:teuthology.orchestra.run.smithi031.stdout: osd objectstore = bluestore 2023-09-07T11:00:50.866 INFO:teuthology.orchestra.run.smithi031.stdout: osd shutdown pgref assert = True 2023-09-07T11:00:50.866 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.867 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr] 2023-09-07T11:00:50.867 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2023-09-07T11:00:50.868 INFO:teuthology.orchestra.run.smithi031.stdout: debug mgr = 20 2023-09-07T11:00:50.868 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2023-09-07T11:00:50.868 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2023-09-07T11:00:50.869 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2023-09-07T11:00:50.869 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2023-09-07T11:00:50.869 INFO:teuthology.orchestra.run.smithi031.stdout: mgr/telemetry/nag = false 2023-09-07T11:00:50.869 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.870 INFO:teuthology.orchestra.run.smithi031.stdout:[mon] 2023-09-07T11:00:50.870 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 0/0 2023-09-07T11:00:50.870 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 0/0 2023-09-07T11:00:50.870 INFO:teuthology.orchestra.run.smithi031.stdout: debug paxos = 0/0 2023-09-07T11:00:50.870 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2023-09-07T11:00:50.872 INFO:teuthology.orchestra.run.smithi031.stdout: mon data avail warn = 5 2023-09-07T11:00:50.873 INFO:teuthology.orchestra.run.smithi031.stdout: mon mgr mkfs grace = 240 2023-09-07T11:00:50.873 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2023-09-07T11:00:50.873 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd reporter subtree level = osd 2023-09-07T11:00:50.874 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd prime pg temp = true 2023-09-07T11:00:50.874 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2023-09-07T11:00:50.874 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.874 INFO:teuthology.orchestra.run.smithi031.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-07T11:00:50.874 INFO:teuthology.orchestra.run.smithi031.stdout: auth mon ticket ttl = 660 # 11m 2023-09-07T11:00:50.875 INFO:teuthology.orchestra.run.smithi031.stdout: auth service ticket ttl = 240 # 4m 2023-09-07T11:00:50.875 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.875 INFO:teuthology.orchestra.run.smithi031.stdout: # don't complain about insecure global_id in the test suite 2023-09-07T11:00:50.875 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-07T11:00:50.875 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-07T11:00:50.875 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.876 INFO:teuthology.orchestra.run.smithi031.stdout: # 1m isn't quite enough 2023-09-07T11:00:50.876 INFO:teuthology.orchestra.run.smithi031.stdout: mon_down_mkfs_grace = 2m 2023-09-07T11:00:50.876 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.877 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_filestore_osds = false 2023-09-07T11:00:50.877 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default crimson = True 2023-09-07T11:00:50.877 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:00:50.877 INFO:teuthology.orchestra.run.smithi031.stdout:[client] 2023-09-07T11:00:50.878 INFO:teuthology.orchestra.run.smithi031.stdout: rgw cache enabled = true 2023-09-07T11:00:50.895 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable ops log = true 2023-09-07T11:00:50.897 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable usage log = true 2023-09-07T11:00:50.897 INFO:teuthology.orchestra.run.smithi031.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-07T11:00:50.897 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-07T11:00:50.898 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.a] 2023-09-07T11:00:50.899 INFO:tasks.ceph:Creating admin key on mon.a... 2023-09-07T11:00:50.900 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-09-07T11:00:51.008 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-07T11:00:51.008 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-07T11:00:51.009 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-09-07T11:00:51.026 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-07T11:00:51.027 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-09-07T11:00:51.081 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi031.front.sepia.ceph.com 2023-09-07T11:00:51.081 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-07T11:00:51.082 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-07T11:00:51.082 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-07T11:00:51.167 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-07T11:00:51.168 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-07T11:00:51.223 INFO:tasks.ceph:Setting up mon nodes... 2023-09-07T11:00:51.223 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-07T11:00:51.224 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-09-07T11:00:51.331 INFO:teuthology.orchestra.run.smithi031.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-09-07T11:00:51.334 INFO:tasks.ceph:Setting up mds nodes... 2023-09-07T11:00:51.335 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-07T11:00:51.335 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-09-07T11:00:51.387 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-09-07T11:00:51.406 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-07T11:00:51.406 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-09-07T11:00:51.407 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-09-07T11:00:51.437 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-07T11:00:51.438 INFO:tasks.ceph:role: osd.0 2023-09-07T11:00:51.438 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com 2023-09-07T11:00:51.438 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-07T11:00:51.515 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-07T11:00:51.515 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-07T11:00:51.515 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-07T11:00:51.516 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-09-07T11:00:51.516 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-07T11:00:51.516 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2023-09-07T11:00:51.516 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-07T11:00:51.516 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-09-07T11:00:51.517 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-07T11:00:51.517 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-07T11:00:52.038 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2023-09-07T11:00:52.044 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2023-09-07T11:00:52.045 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-09-07T11:00:52.199 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-09-07T11:00:52.238 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-09-07T11:00:52.314 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-07T11:00:52.314 INFO:tasks.ceph:role: osd.1 2023-09-07T11:00:52.314 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com 2023-09-07T11:00:52.315 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-07T11:00:52.392 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-07T11:00:52.393 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-07T11:00:52.393 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-07T11:00:52.393 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-09-07T11:00:52.393 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-07T11:00:52.394 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2023-09-07T11:00:52.394 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-07T11:00:52.394 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-09-07T11:00:52.394 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-07T11:00:52.394 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-07T11:00:52.837 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2023-09-07T11:00:52.839 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2023-09-07T11:00:52.839 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-09-07T11:00:52.930 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-09-07T11:00:52.967 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-09-07T11:00:53.038 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-07T11:00:53.038 INFO:tasks.ceph:role: osd.2 2023-09-07T11:00:53.039 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com 2023-09-07T11:00:53.039 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-07T11:00:53.116 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-07T11:00:53.116 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-07T11:00:53.117 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-07T11:00:53.117 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-09-07T11:00:53.117 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-07T11:00:53.117 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2023-09-07T11:00:53.117 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-07T11:00:53.118 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-09-07T11:00:53.118 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-07T11:00:53.118 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-07T11:00:53.598 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2023-09-07T11:00:53.603 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2023-09-07T11:00:53.604 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-09-07T11:00:53.735 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-09-07T11:00:53.773 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-07T11:00:54.332 INFO:teuthology.orchestra.run.smithi031.stderr:WARNING: debug mode. Not for benchmarking or production 2023-09-07T11:00:54.333 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:00:54,333 seastar - Reactor backend: linux-aio 2023-09-07T11:00:54.353 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:00:54,349 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-09-07T11:00:54.353 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:00:54,349 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-09-07T11:00:54.353 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:00:54,349 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-09-07T11:00:54.354 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:00:54,349 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-09-07T11:00:54.354 INFO:teuthology.orchestra.run.smithi031.stderr:==27642==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-09-07T11:00:54.362 INFO:teuthology.orchestra.run.smithi031.stderr:==27642==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff41440000; bottom 0x7f4d44328000; size: 0x00b1fd118000 (764454993920) 2023-09-07T11:00:54.363 INFO:teuthology.orchestra.run.smithi031.stderr:False positive error reports may follow 2023-09-07T11:00:54.363 INFO:teuthology.orchestra.run.smithi031.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-09-07T11:00:54.389 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c0db7f0 0x2c0dcb9e 0x2c0de0ec 0x2c0de470 0x2c0de5ba 0x2c0dea10 0x54daf 0x22914734 0x228f3606 0x228f3ca8 0x228fbcc4 0x22907fa5 0x229362e6 0x2291afca 0x2291e564 0x2291eacb 0x22963e89 0x229646f5 0x229013ff 0x2291084c 0x22911f2b 0x2296988a 0x2296a950 0x22926762 0x2296d692 0x2296f5c1 0x22970fc6 0x22971d24 0x228eda8c 0x227b532c 0x2263970e 0x245e3321 0x245e8a49 0x245eafe4 0x245ec1bd 0x2219062f 0x221dad7c 0x222057e7 0x2c07dc31 0x2c11b918 0x2c33f6e6 0x2c341531 0x2bd99f50 0x2bd9be51 0x1e9f3010 0x1e9fd17d 0x1e87da37 0x3feaf 0x3ff5f 0x1e339a04 2023-09-07T11:00:54.389 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 2023-09-07T11:00:55.017 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-09-07T11:00:55.018 INFO:teuthology.orchestra.run.smithi031.stderr:Suppressions used: 2023-09-07T11:00:55.018 INFO:teuthology.orchestra.run.smithi031.stderr: count bytes template 2023-09-07T11:00:55.018 INFO:teuthology.orchestra.run.smithi031.stderr: 1 8 MallocExtension::Register 2023-09-07T11:00:55.018 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-09-07T11:00:55.019 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-09-07T11:00:55.069 INFO:teuthology.orchestra.run.smithi031.stderr:WARNING: debug mode. Not for benchmarking or production 2023-09-07T11:00:55.070 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:00:55,070 seastar - Reactor backend: linux-aio 2023-09-07T11:00:55.088 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:00:55,088 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-09-07T11:00:55.088 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:00:55,088 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-09-07T11:00:55.088 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:00:55,088 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-09-07T11:00:55.095 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:00:55,095 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-09-07T11:00:55.097 INFO:teuthology.orchestra.run.smithi031.stderr:==27640==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-09-07T11:00:55.104 INFO:teuthology.orchestra.run.smithi031.stderr:==27640==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fff41440000; bottom 0x7f4d42111000; size: 0x00b1ff32f000 (764490739712) 2023-09-07T11:00:55.104 INFO:teuthology.orchestra.run.smithi031.stderr:False positive error reports may follow 2023-09-07T11:00:55.104 INFO:teuthology.orchestra.run.smithi031.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-09-07T11:00:55.130 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c0db7f0 0x2c0dcb9e 0x2c0de0ec 0x2c0de470 0x2c0de5ba 0x2c0dea10 0x54daf 0x2255df2f 0x2259ddde 0x225ecd46 0x225fa897 0x226008e3 0x226022a2 0x245e3dee 0x245e8a49 0x245eafe4 0x245ec1bd 0x2219062f 0x221dad7c 0x222057e7 0x2c07dc31 0x2c11b918 0x2c33f6e6 0x2c341531 0x2bd99f50 0x2bd9be51 0x1e87f2ed 0x3feaf 0x3ff5f 0x1e339a04 2023-09-07T11:00:55.130 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 2023-09-07T11:00:55.191 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c0db7f0 0x2c0dcb9e 0x2c0de0ec 0x2c0de470 0x2c0de5ba 0x2c0dea10 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x25dbf99e 0x25dc1fe7 0x25dc3df8 0x25dc665d 0x25a6f277 0x1e938aa4 0x1e8ada11 0x1e8b197d 0x2c45f411 2023-09-07T11:00:55.191 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffbb654847 0xffffffffbb650167 0xffffffffbb8000aa 2023-09-07T11:00:59.138 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-09-07T11:00:59.138 INFO:teuthology.orchestra.run.smithi031.stderr:Suppressions used: 2023-09-07T11:00:59.138 INFO:teuthology.orchestra.run.smithi031.stderr: count bytes template 2023-09-07T11:00:59.138 INFO:teuthology.orchestra.run.smithi031.stderr: 1 8 MallocExtension::Register 2023-09-07T11:00:59.139 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-09-07T11:00:59.139 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-09-07T11:00:59.139 INFO:teuthology.orchestra.run.smithi031.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid e8f5877e-ee15-4b77-b5d2-4d38002bcd0c 2023-09-07T11:00:59.175 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-09-07T11:00:59.206 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-07T11:00:59.751 INFO:teuthology.orchestra.run.smithi031.stderr:WARNING: debug mode. Not for benchmarking or production 2023-09-07T11:00:59.752 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:00:59,749 seastar - Reactor backend: linux-aio 2023-09-07T11:00:59.769 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:00:59,765 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-09-07T11:00:59.769 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:00:59,765 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-09-07T11:00:59.770 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:00:59,765 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-09-07T11:00:59.770 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:00:59,765 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-09-07T11:00:59.772 INFO:teuthology.orchestra.run.smithi031.stderr:==27824==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-09-07T11:00:59.773 INFO:teuthology.orchestra.run.smithi031.stderr:==27824==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe68727000; bottom 0x7ff603323000; size: 0x000865404000 (36058447872) 2023-09-07T11:00:59.773 INFO:teuthology.orchestra.run.smithi031.stderr:False positive error reports may follow 2023-09-07T11:00:59.774 INFO:teuthology.orchestra.run.smithi031.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-09-07T11:00:59.799 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c0db7f0 0x2c0dcb9e 0x2c0de0ec 0x2c0de470 0x2c0de5ba 0x2c0dea10 0x54daf 0x225c78bb 0x226eee2a 0x2260fff5 0x22639efe 0x245e3321 0x245e8a49 0x245eafe4 0x245ec1bd 0x2219062f 0x221dad7c 0x222057e7 0x2c07dc31 0x2c11b918 0x2c33f6e6 0x2c341531 0x2bd99f50 0x2bd9be51 0x1e9f3010 0x1e9fd17d 0x1e87da37 0x3feaf 0x3ff5f 0x1e339a04 2023-09-07T11:00:59.800 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 2023-09-07T11:01:00.311 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-09-07T11:01:00.314 INFO:teuthology.orchestra.run.smithi031.stderr:Suppressions used: 2023-09-07T11:01:00.314 INFO:teuthology.orchestra.run.smithi031.stderr: count bytes template 2023-09-07T11:01:00.315 INFO:teuthology.orchestra.run.smithi031.stderr: 1 8 MallocExtension::Register 2023-09-07T11:01:00.315 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-09-07T11:01:00.315 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-09-07T11:01:00.361 INFO:teuthology.orchestra.run.smithi031.stderr:WARNING: debug mode. Not for benchmarking or production 2023-09-07T11:01:00.362 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:01:00,362 seastar - Reactor backend: linux-aio 2023-09-07T11:01:00.381 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:01:00,380 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-09-07T11:01:00.381 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:01:00,380 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-09-07T11:01:00.381 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:01:00,380 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-09-07T11:01:00.387 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:01:00,388 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-09-07T11:01:00.390 INFO:teuthology.orchestra.run.smithi031.stderr:==27822==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-09-07T11:01:00.396 INFO:teuthology.orchestra.run.smithi031.stderr:==27822==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe68727000; bottom 0x7ff60110c000; size: 0x00086761b000 (36094193664) 2023-09-07T11:01:00.396 INFO:teuthology.orchestra.run.smithi031.stderr:False positive error reports may follow 2023-09-07T11:01:00.396 INFO:teuthology.orchestra.run.smithi031.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-09-07T11:01:00.422 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c0db7f0 0x2c0dcb9e 0x2c0de0ec 0x2c0de470 0x2c0de5ba 0x2c0dea10 0x54daf 0x1e6e7ef8 0x1e7c17c7 0x226009c8 0x226022a2 0x245e3dee 0x245e8a49 0x245eafe4 0x245ec1bd 0x2219062f 0x221dad7c 0x222057e7 0x2c07dc31 0x2c11b918 0x2c33f6e6 0x2c341531 0x2bd99f50 0x2bd9be51 0x1e87f2ed 0x3feaf 0x3ff5f 0x1e339a04 2023-09-07T11:01:00.422 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 2023-09-07T11:01:04.443 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-09-07T11:01:04.443 INFO:teuthology.orchestra.run.smithi031.stderr:Suppressions used: 2023-09-07T11:01:04.444 INFO:teuthology.orchestra.run.smithi031.stderr: count bytes template 2023-09-07T11:01:04.444 INFO:teuthology.orchestra.run.smithi031.stderr: 1 8 MallocExtension::Register 2023-09-07T11:01:04.444 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-09-07T11:01:04.445 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-09-07T11:01:04.445 INFO:teuthology.orchestra.run.smithi031.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid e8f5877e-ee15-4b77-b5d2-4d38002bcd0c 2023-09-07T11:01:04.495 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-09-07T11:01:04.527 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-07T11:01:05.067 INFO:teuthology.orchestra.run.smithi031.stderr:WARNING: debug mode. Not for benchmarking or production 2023-09-07T11:01:05.068 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:01:05,068 seastar - Reactor backend: linux-aio 2023-09-07T11:01:05.086 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:01:05,084 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-09-07T11:01:05.087 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:01:05,084 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-09-07T11:01:05.087 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:01:05,084 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-09-07T11:01:05.087 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:01:05,084 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-09-07T11:01:05.087 INFO:teuthology.orchestra.run.smithi031.stderr:==28021==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-09-07T11:01:05.095 INFO:teuthology.orchestra.run.smithi031.stderr:==28021==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc2082f000; bottom 0x7f9746724000; size: 0x0064da10b000 (433155256320) 2023-09-07T11:01:05.095 INFO:teuthology.orchestra.run.smithi031.stderr:False positive error reports may follow 2023-09-07T11:01:05.095 INFO:teuthology.orchestra.run.smithi031.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-09-07T11:01:05.118 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c0db7f0 0x2c0dcb9e 0x2c0de0ec 0x2c0de470 0x2c0de5ba 0x2c0dea10 0x54daf 0x1e60526a 0x227390ec 0x2273943c 0x2273a9ef 0x2273943c 0x2273943c 0x228a2302 0x22736825 0x2296b657 0x2296d97c 0x2296f5c1 0x22970fc6 0x22971d24 0x228eda8c 0x227b532c 0x2263970e 0x245e3321 0x245e8a49 0x245eafe4 0x245ec1bd 0x2219062f 0x221dad7c 0x222057e7 0x2c07dc31 0x2c11b918 0x2c33f6e6 0x2c341531 0x2bd99f50 0x2bd9be51 0x1e9f3010 0x1e9fd17d 0x1e87da37 0x3feaf 0x3ff5f 0x1e339a04 2023-09-07T11:01:05.119 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 2023-09-07T11:01:05.644 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-09-07T11:01:05.644 INFO:teuthology.orchestra.run.smithi031.stderr:Suppressions used: 2023-09-07T11:01:05.644 INFO:teuthology.orchestra.run.smithi031.stderr: count bytes template 2023-09-07T11:01:05.644 INFO:teuthology.orchestra.run.smithi031.stderr: 1 8 MallocExtension::Register 2023-09-07T11:01:05.645 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-09-07T11:01:05.645 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-09-07T11:01:05.706 INFO:teuthology.orchestra.run.smithi031.stderr:WARNING: debug mode. Not for benchmarking or production 2023-09-07T11:01:05.706 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:01:05,707 seastar - Reactor backend: linux-aio 2023-09-07T11:01:05.725 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:01:05,725 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-09-07T11:01:05.725 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:01:05,725 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-09-07T11:01:05.725 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:01:05,725 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-09-07T11:01:05.732 INFO:teuthology.orchestra.run.smithi031.stderr:INFO 2023-09-07 11:01:05,732 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-09-07T11:01:05.735 INFO:teuthology.orchestra.run.smithi031.stderr:==28019==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-09-07T11:01:05.741 INFO:teuthology.orchestra.run.smithi031.stderr:==28019==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc2082f000; bottom 0x7f974450d000; size: 0x0064dc322000 (433191002112) 2023-09-07T11:01:05.741 INFO:teuthology.orchestra.run.smithi031.stderr:False positive error reports may follow 2023-09-07T11:01:05.741 INFO:teuthology.orchestra.run.smithi031.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-09-07T11:01:05.767 INFO:teuthology.orchestra.run.smithi031.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c0db7f0 0x2c0dcb9e 0x2c0de0ec 0x2c0de470 0x2c0de5ba 0x2c0dea10 0x54daf 0x3973f 0x94437 0x94d46 0x2260296c 0x245e3dee 0x245e8a49 0x245eafe4 0x245ec1bd 0x2219062f 0x221dad7c 0x222057e7 0x2c07dc31 0x2c11b918 0x2c33f6e6 0x2c341531 0x2bd99f50 0x2bd9be51 0x1e87f2ed 0x3feaf 0x3ff5f 0x1e339a04 2023-09-07T11:01:05.767 INFO:teuthology.orchestra.run.smithi031.stderr:kernel callstack: 2023-09-07T11:01:09.791 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-09-07T11:01:09.791 INFO:teuthology.orchestra.run.smithi031.stderr:Suppressions used: 2023-09-07T11:01:09.791 INFO:teuthology.orchestra.run.smithi031.stderr: count bytes template 2023-09-07T11:01:09.792 INFO:teuthology.orchestra.run.smithi031.stderr: 1 8 MallocExtension::Register 2023-09-07T11:01:09.792 INFO:teuthology.orchestra.run.smithi031.stderr:----------------------------------------------------- 2023-09-07T11:01:09.792 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-09-07T11:01:09.792 INFO:teuthology.orchestra.run.smithi031.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid e8f5877e-ee15-4b77-b5d2-4d38002bcd0c 2023-09-07T11:01:09.824 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-09-07T11:01:09.857 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-07T11:01:09.858 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-07T11:01:09.858 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-09-07T11:01:09.928 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-07T11:01:09.928 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-09-07T11:01:10.001 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-07T11:01:10.001 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-09-07T11:01:10.072 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-07T11:01:10.073 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-09-07T11:01:10.143 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-07T11:01:10.143 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-09-07T11:01:10.198 INFO:tasks.ceph:Adding keys to all mons... 2023-09-07T11:01:10.199 DEBUG:teuthology.orchestra.run.smithi031:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-07T11:01:10.267 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr.x] 2023-09-07T11:01:10.267 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQBjrflk5gnHExAAOU+GfnDhl/GkDdhQl3uiaQ== 2023-09-07T11:01:10.268 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.0] 2023-09-07T11:01:10.268 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQBnrflkV70/CRAAic0tALcqb3iV/xQ9JviVtQ== 2023-09-07T11:01:10.268 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.1] 2023-09-07T11:01:10.268 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQBsrflk0/SLGhAAQrRvk+OJrG0S19rWASBcig== 2023-09-07T11:01:10.268 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.2] 2023-09-07T11:01:10.269 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQBxrflkr8waLxAAfdbOIP2P/8NonNUTZAPzbA== 2023-09-07T11:01:10.269 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2023-09-07T11:01:10.269 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQBjrflkQ2oaFxAAc6VpGZW/d51rJdjytY7A3A== 2023-09-07T11:01:10.270 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-07T11:01:10.373 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-07T11:01:10.437 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-07T11:01:10.499 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-07T11:01:10.561 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-07T11:01:10.626 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-07T11:01:10.626 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-09-07T11:01:10.657 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-07T11:01:10.761 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:10.761+0000 7f050b228980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:10.762 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:10.761+0000 7f050b228980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:10.781 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-09-07T11:01:10.812 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-07T11:01:10.867 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-09-07T11:01:10.868 INFO:tasks.ceph.mon.a:Restarting daemon 2023-09-07T11:01:10.868 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-09-07T11:01:10.909 INFO:tasks.ceph.mon.a:Started 2023-09-07T11:01:10.910 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-09-07T11:01:10.911 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-09-07T11:01:10.911 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-09-07T11:01:10.913 INFO:tasks.ceph.mgr.x:Started 2023-09-07T11:01:10.914 INFO:tasks.ceph:Setting crush tunables to default 2023-09-07T11:01:10.914 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd crush tunables default 2023-09-07T11:01:10.997 INFO:tasks.ceph.mon.a.smithi031.stderr:2023-09-07T11:01:10.996+0000 7f5deb617980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:10.997 INFO:tasks.ceph.mon.a.smithi031.stderr:2023-09-07T11:01:10.996+0000 7f5deb617980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:10.997 INFO:tasks.ceph.mon.a.smithi031.stderr:2023-09-07T11:01:10.996+0000 7f5deb617980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:10.998 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:10.997+0000 7fc486b6bf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:11.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:11.025+0000 7f3d8375f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:11.056 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:11.056+0000 7f3d8375f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:11.128 INFO:teuthology.orchestra.run.smithi031.stderr:adjusted tunables profile to default 2023-09-07T11:01:11.139 INFO:tasks.ceph:check_enable_crimson: True 2023-09-07T11:01:11.139 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2023-09-07T11:01:11.140 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2023-09-07T11:01:11.202 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:11.201+0000 7fc486b6bf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:11.203 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:11.201+0000 7fc486b6bf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:11.250 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:11.245+0000 7fd7d8905640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:11.252 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:11.251+0000 7fd7d8905640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:11.336 INFO:tasks.ceph.mon.a.smithi031.stderr:2023-09-07T11:01:11.334+0000 7f5de35f1640 -1 WARNING: experimental feature 'crimson' is enabled 2023-09-07T11:01:11.337 INFO:tasks.ceph.mon.a.smithi031.stderr:Please be aware that this feature is experimental, untested, 2023-09-07T11:01:11.337 INFO:tasks.ceph.mon.a.smithi031.stderr:unsupported, and may result in data corruption, data loss, 2023-09-07T11:01:11.337 INFO:tasks.ceph.mon.a.smithi031.stderr:and/or irreparable damage to your cluster. Do not use 2023-09-07T11:01:11.337 INFO:tasks.ceph.mon.a.smithi031.stderr:feature with important data. 2023-09-07T11:01:11.337 INFO:tasks.ceph.mon.a.smithi031.stderr: 2023-09-07T11:01:11.691 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:11.690+0000 7fc486b6bf00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-07T11:01:11.744 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:11.743+0000 7fc486b6bf00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-07T11:01:12.140 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-09-07T11:01:12.141 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-07T11:01:12.142 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-09-07T11:01:12.175 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-07T11:01:12.176 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-09-07T11:01:12.247 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-07T11:01:12.247 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-09-07T11:01:12.310 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:12.310+0000 7fc486b6bf00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-07T11:01:12.319 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd new 09dc373e-7f2d-4b5b-9fee-16b40027197d 0 2023-09-07T11:01:12.365 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:12.365+0000 7fc486b6bf00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-07T11:01:12.413 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:12.412+0000 7fc486b6bf00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-07T11:01:12.470 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:12.464+0000 7fab9571f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:12.471 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:12.471+0000 7fc486b6bf00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-07T11:01:12.472 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:12.471+0000 7fab9571f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:12.517 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:12.516+0000 7fc486b6bf00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-07T11:01:12.553 INFO:teuthology.orchestra.run.smithi031.stdout:0 2023-09-07T11:01:12.564 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd new 6a75809b-e220-4c7d-a142-2de2f15da9eb 1 2023-09-07T11:01:12.626 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:12.623+0000 7fc486b6bf00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-07T11:01:12.699 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:12.699+0000 7fc486b6bf00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-07T11:01:12.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:12.710+0000 7f699ca57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:12.718 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:12.717+0000 7f699ca57640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:12.749 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:12.749+0000 7fc486b6bf00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-07T11:01:12.799 INFO:teuthology.orchestra.run.smithi031.stdout:1 2023-09-07T11:01:12.804 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:12.804+0000 7fc486b6bf00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-07T11:01:12.810 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd new db102e2a-75bf-47af-b38b-cd7cbfb862b2 2 2023-09-07T11:01:12.855 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:12.853+0000 7fc486b6bf00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-07T11:01:12.961 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:12.955+0000 7fe8ef31f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:12.963 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:12.962+0000 7fe8ef31f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:13.042 INFO:teuthology.orchestra.run.smithi031.stdout:2 2023-09-07T11:01:13.053 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2023-09-07T11:01:13.131 INFO:tasks.ceph.osd.0:Restarting daemon 2023-09-07T11:01:13.132 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2023-09-07T11:01:13.173 INFO:tasks.ceph.osd.0:Started 2023-09-07T11:01:13.174 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2023-09-07T11:01:13.210 INFO:tasks.ceph.osd.1:Restarting daemon 2023-09-07T11:01:13.210 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2023-09-07T11:01:13.254 INFO:tasks.ceph.osd.1:Started 2023-09-07T11:01:13.254 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2023-09-07T11:01:13.266 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:13.265+0000 7fc486b6bf00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-07T11:01:13.295 INFO:tasks.ceph.osd.2:Restarting daemon 2023-09-07T11:01:13.295 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2023-09-07T11:01:13.340 INFO:tasks.ceph.osd.2:Started 2023-09-07T11:01:13.340 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-07T11:01:13.429 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:13.428+0000 7fc486b6bf00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-07T11:01:13.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:13.557+0000 7f5595220640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:13.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:13.568+0000 7f5595220640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:13.594 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:13.591+0000 7fc486b6bf00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-07T11:01:13.651 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:13.648+0000 7fc486b6bf00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-07T11:01:13.708 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:01:13.709 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":6,"fsid":"e8f5877e-ee15-4b77-b5d2-4d38002bcd0c","created":"2023-09-07T11:01:11.042792+0000","modified":"2023-09-07T11:01:13.040145+0000","last_up_change":"0.000000","last_in_change":"2023-09-07T11:01:13.040145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"09dc373e-7f2d-4b5b-9fee-16b40027197d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6a75809b-e220-4c7d-a142-2de2f15da9eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"db102e2a-75bf-47af-b38b-cd7cbfb862b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-07T11:01:13.735 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-09-07T11:01:13.736 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-07T11:01:13.765 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:13.764+0000 7fc486b6bf00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-07T11:01:13.873 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:13.870+0000 7fc486b6bf00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-07T11:01:14.019 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:14.019+0000 7fc486b6bf00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-07T11:01:14.038 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-07T11:01:14.065 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:14.064+0000 7fc486b6bf00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-07T11:01:14.139 INFO:teuthology.misc.health.smithi031.stderr:2023-09-07T11:01:14.134+0000 7f47a2260640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:14.141 INFO:teuthology.misc.health.smithi031.stderr:2023-09-07T11:01:14.141+0000 7f47a2260640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:14.225 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:14.221+0000 7fc486b6bf00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-07T11:01:14.267 INFO:teuthology.misc.health.smithi031.stdout: 2023-09-07T11:01:14.267 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":6,"fsid":"e8f5877e-ee15-4b77-b5d2-4d38002bcd0c","created":"2023-09-07T11:01:11.042792+0000","modified":"2023-09-07T11:01:13.040145+0000","last_up_change":"0.000000","last_in_change":"2023-09-07T11:01:13.040145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"09dc373e-7f2d-4b5b-9fee-16b40027197d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6a75809b-e220-4c7d-a142-2de2f15da9eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"db102e2a-75bf-47af-b38b-cd7cbfb862b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-07T11:01:14.282 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-09-07T11:01:15.089 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:15.087+0000 7fc486b6bf00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-07T11:01:15.378 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:15.374+0000 7fc486b6bf00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-07T11:01:15.429 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:15.427+0000 7fc486b6bf00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-07T11:01:15.543 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:15.539+0000 7fc486b6bf00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-07T11:01:16.563 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-09-07T11:01:16.561+0000 7fc4267c2640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-09-07T11:01:20.588 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-07T11:01:20.698 INFO:teuthology.misc.health.smithi031.stderr:2023-09-07T11:01:20.694+0000 7fdaf00ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:20.702 INFO:teuthology.misc.health.smithi031.stderr:2023-09-07T11:01:20.699+0000 7fdaf00ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:20.887 INFO:teuthology.misc.health.smithi031.stdout: 2023-09-07T11:01:20.887 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":9,"fsid":"e8f5877e-ee15-4b77-b5d2-4d38002bcd0c","created":"2023-09-07T11:01:11.042792+0000","modified":"2023-09-07T11:01:20.572656+0000","last_up_change":"2023-09-07T11:01:18.567982+0000","last_in_change":"2023-09-07T11:01:13.040145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-07T11:01:19.614675+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"09dc373e-7f2d-4b5b-9fee-16b40027197d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3351905862}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":3351905862}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":3351905862}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6813","nonce":3351905862}]},"public_addr":"172.21.15.31:6806/3351905862","cluster_addr":"172.21.15.31:6807/3351905862","heartbeat_back_addr":"172.21.15.31:6812/3351905862","heartbeat_front_addr":"172.21.15.31:6813/3351905862","state":["exists","up"]},{"osd":1,"uuid":"6a75809b-e220-4c7d-a142-2de2f15da9eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2086429254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":2086429254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":2086429254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2086429254}]},"public_addr":"172.21.15.31:6802/2086429254","cluster_addr":"172.21.15.31:6803/2086429254","heartbeat_back_addr":"172.21.15.31:6809/2086429254","heartbeat_front_addr":"172.21.15.31:6808/2086429254","state":["exists","up"]},{"osd":2,"uuid":"db102e2a-75bf-47af-b38b-cd7cbfb862b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2054859839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":2054859839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2054859839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":2054859839}]},"public_addr":"172.21.15.31:6804/2054859839","cluster_addr":"172.21.15.31:6805/2054859839","heartbeat_back_addr":"172.21.15.31:6810/2054859839","heartbeat_front_addr":"172.21.15.31:6811/2054859839","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-07T11:01:20.898 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-09-07T11:01:20.898 INFO:tasks.ceph:Creating RBD pool 2023-09-07T11:01:20.898 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd pool create rbd 8 2023-09-07T11:01:21.011 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:21.004+0000 7f24acd9f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:21.015 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:21.013+0000 7f24acd9f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:21.582 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'rbd' created 2023-09-07T11:01:21.597 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-07T11:01:21.721 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:21.711+0000 7febd88d6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:21.723 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:21.722+0000 7febd88d6640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:22.588 INFO:teuthology.orchestra.run.smithi031.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-07T11:01:22.601 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-09-07T11:01:22.602 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-09-07T11:01:22.602 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-09-07T11:01:22.603 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-09-07T11:01:22.603 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-07T11:01:22.734 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:22.725+0000 7f29a6edf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:22.747 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:22.731+0000 7f29a6edf640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:22.971 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:01:22.983 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":682726219},{"type":"v1","addr":"172.21.15.31:6801","nonce":682726219}]},"active_addr":"172.21.15.31:6801/682726219","active_change":"2023-09-07T11:01:15.554625+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":4207296803}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":2650251729}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":438659042}]}]} 2023-09-07T11:01:22.985 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-09-07T11:01:22.986 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-09-07T11:01:22.986 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-07T11:01:23.103 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:23.098+0000 7f5d88f9f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:23.105 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:23.103+0000 7f5d88f9f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:23.289 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:01:23.290 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":11,"fsid":"e8f5877e-ee15-4b77-b5d2-4d38002bcd0c","created":"2023-09-07T11:01:11.042792+0000","modified":"2023-09-07T11:01:22.583525+0000","last_up_change":"2023-09-07T11:01:18.567982+0000","last_in_change":"2023-09-07T11:01:13.040145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-07T11:01:19.614675+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-07T11:01:21.226942+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"09dc373e-7f2d-4b5b-9fee-16b40027197d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3351905862}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":3351905862}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":3351905862}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6813","nonce":3351905862}]},"public_addr":"172.21.15.31:6806/3351905862","cluster_addr":"172.21.15.31:6807/3351905862","heartbeat_back_addr":"172.21.15.31:6812/3351905862","heartbeat_front_addr":"172.21.15.31:6813/3351905862","state":["exists","up"]},{"osd":1,"uuid":"6a75809b-e220-4c7d-a142-2de2f15da9eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2086429254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":2086429254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":2086429254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2086429254}]},"public_addr":"172.21.15.31:6802/2086429254","cluster_addr":"172.21.15.31:6803/2086429254","heartbeat_back_addr":"172.21.15.31:6809/2086429254","heartbeat_front_addr":"172.21.15.31:6808/2086429254","state":["exists","up"]},{"osd":2,"uuid":"db102e2a-75bf-47af-b38b-cd7cbfb862b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2054859839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":2054859839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2054859839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":2054859839}]},"public_addr":"172.21.15.31:6804/2054859839","cluster_addr":"172.21.15.31:6805/2054859839","heartbeat_back_addr":"172.21.15.31:6810/2054859839","heartbeat_front_addr":"172.21.15.31:6811/2054859839","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-07T11:01:23.301 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-09-07T11:01:23.301 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-07T11:01:23.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:23.408+0000 7f55242ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:23.416 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:23.413+0000 7f55242ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:23.605 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:01:23.605 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":12,"fsid":"e8f5877e-ee15-4b77-b5d2-4d38002bcd0c","created":"2023-09-07T11:01:11.042792+0000","modified":"2023-09-07T11:01:23.586915+0000","last_up_change":"2023-09-07T11:01:18.567982+0000","last_in_change":"2023-09-07T11:01:13.040145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-07T11:01:19.614675+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-07T11:01:21.226942+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"09dc373e-7f2d-4b5b-9fee-16b40027197d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3351905862}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6807","nonce":3351905862}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":3351905862}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6813","nonce":3351905862}]},"public_addr":"172.21.15.31:6806/3351905862","cluster_addr":"172.21.15.31:6807/3351905862","heartbeat_back_addr":"172.21.15.31:6812/3351905862","heartbeat_front_addr":"172.21.15.31:6813/3351905862","state":["exists","up"]},{"osd":1,"uuid":"6a75809b-e220-4c7d-a142-2de2f15da9eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2086429254}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6803","nonce":2086429254}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6809","nonce":2086429254}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2086429254}]},"public_addr":"172.21.15.31:6802/2086429254","cluster_addr":"172.21.15.31:6803/2086429254","heartbeat_back_addr":"172.21.15.31:6809/2086429254","heartbeat_front_addr":"172.21.15.31:6808/2086429254","state":["exists","up"]},{"osd":2,"uuid":"db102e2a-75bf-47af-b38b-cd7cbfb862b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2054859839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6805","nonce":2054859839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2054859839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6811","nonce":2054859839}]},"public_addr":"172.21.15.31:6804/2054859839","cluster_addr":"172.21.15.31:6805/2054859839","heartbeat_back_addr":"172.21.15.31:6810/2054859839","heartbeat_front_addr":"172.21.15.31:6811/2054859839","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-07T11:01:23.619 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-09-07T11:01:23.619 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-09-07T11:01:23.620 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-09-07T11:01:23.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:23.732+0000 7f53650ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:23.738 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:23.732+0000 7f4890857640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:23.739 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:23.733+0000 7fa3a42ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:23.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:23.739+0000 7f53650ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:23.749 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:23.739+0000 7f4890857640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:23.750 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:23.739+0000 7fa3a42ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:23.767 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-09-07T11:01:23.768 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-07T11:01:23.772 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-09-07T11:01:23.772 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-07T11:01:23.774 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-09-07T11:01:23.775 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-07T11:01:23.893 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:23.885+0000 7fdd5291f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:23.896 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:23.894+0000 7fdd5291f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:23.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:23.926+0000 7f188f40f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:23.933 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:23.932+0000 7f188f40f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:23.939 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:23.933+0000 7f8417d1f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:23.946 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:23.944+0000 7f8417d1f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:24.089 INFO:teuthology.orchestra.run.smithi031.stdout:34359738372 2023-09-07T11:01:24.102 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2023-09-07T11:01:24.124 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-09-07T11:01:24.135 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-09-07T11:01:24.136 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2023-09-07T11:01:24.136 DEBUG:teuthology.parallel:result is None 2023-09-07T11:01:24.147 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2023-09-07T11:01:24.147 DEBUG:teuthology.parallel:result is None 2023-09-07T11:01:25.103 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-07T11:01:25.234 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:25.229+0000 7f7cbdd9f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:25.236 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:25.235+0000 7f7cbdd9f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:25.419 INFO:teuthology.orchestra.run.smithi031.stdout:34359738372 2023-09-07T11:01:25.434 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2023-09-07T11:01:26.436 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-07T11:01:26.563 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:26.557+0000 7f811926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:26.569 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:26.564+0000 7f811926d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:26.747 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-09-07T11:01:26.759 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2023-09-07T11:01:26.760 DEBUG:teuthology.parallel:result is None 2023-09-07T11:01:26.760 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-07T11:01:26.760 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-07T11:01:26.872 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:26.868+0000 7fbbb3f9f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:26.873 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:26.873+0000 7fbbb3f9f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:27.053 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:01:27.054 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-09-07T11:01:27.064 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-09-07T11:01:25.559276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80760,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034760,"statfs":{"total":289910292480,"available":289827594240,"internally_reserved":0,"allocated":270336,"data_stored":1224755,"data_compressed":59920,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.957248"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-07T11:01:22.718489+0000","last_change":"2023-09-07T11:01:22.718599+0000","last_active":"2023-09-07T11:01:22.718489+0000","last_peered":"2023-09-07T11:01:22.718489+0000","last_clean":"2023-09-07T11:01:21.579328+0000","last_became_active":"2023-09-07T11:01:22.714692+0000","last_became_peered":"2023-09-07T11:01:22.714692+0000","last_unstale":"2023-09-07T11:01:22.718489+0000","last_undegraded":"2023-09-07T11:01:22.718489+0000","last_fullsized":"2023-09-07T11:01:22.718489+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_clean_scrub_stamp":"2023-09-07T11:01:21.579328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-07T11:01:22.704476+0000","last_change":"2023-09-07T11:01:22.704528+0000","last_active":"2023-09-07T11:01:22.704476+0000","last_peered":"2023-09-07T11:01:22.704476+0000","last_clean":"2023-09-07T11:01:21.579328+0000","last_became_active":"2023-09-07T11:01:22.702754+0000","last_became_peered":"2023-09-07T11:01:22.702754+0000","last_unstale":"2023-09-07T11:01:22.704476+0000","last_undegraded":"2023-09-07T11:01:22.704476+0000","last_fullsized":"2023-09-07T11:01:22.704476+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_clean_scrub_stamp":"2023-09-07T11:01:21.579328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-07T11:01:22.674342+0000","last_change":"2023-09-07T11:01:22.674524+0000","last_active":"2023-09-07T11:01:22.674342+0000","last_peered":"2023-09-07T11:01:22.674342+0000","last_clean":"2023-09-07T11:01:21.579328+0000","last_became_active":"2023-09-07T11:01:22.655450+0000","last_became_peered":"2023-09-07T11:01:22.655450+0000","last_unstale":"2023-09-07T11:01:22.674342+0000","last_undegraded":"2023-09-07T11:01:22.674342+0000","last_fullsized":"2023-09-07T11:01:22.674342+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_clean_scrub_stamp":"2023-09-07T11:01:21.579328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-07T11:01:22.704653+0000","last_change":"2023-09-07T11:01:22.704716+0000","last_active":"2023-09-07T11:01:22.704653+0000","last_peered":"2023-09-07T11:01:22.704653+0000","last_clean":"2023-09-07T11:01:21.579328+0000","last_became_active":"2023-09-07T11:01:22.702322+0000","last_became_peered":"2023-09-07T11:01:22.702322+0000","last_unstale":"2023-09-07T11:01:22.704653+0000","last_undegraded":"2023-09-07T11:01:22.704653+0000","last_fullsized":"2023-09-07T11:01:22.704653+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_clean_scrub_stamp":"2023-09-07T11:01:21.579328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-07T11:01:23.621725+0000","last_change":"2023-09-07T11:01:22.694884+0000","last_active":"2023-09-07T11:01:23.621725+0000","last_peered":"2023-09-07T11:01:23.621725+0000","last_clean":"2023-09-07T11:01:23.621725+0000","last_became_active":"2023-09-07T11:01:22.692860+0000","last_became_peered":"2023-09-07T11:01:22.692860+0000","last_unstale":"2023-09-07T11:01:23.621725+0000","last_undegraded":"2023-09-07T11:01:23.621725+0000","last_fullsized":"2023-09-07T11:01:23.621725+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_clean_scrub_stamp":"2023-09-07T11:01:21.579328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-07T11:01:23.596771+0000","last_change":"2023-09-07T11:01:22.688511+0000","last_active":"2023-09-07T11:01:23.596771+0000","last_peered":"2023-09-07T11:01:23.596771+0000","last_clean":"2023-09-07T11:01:23.596771+0000","last_became_active":"2023-09-07T11:01:22.686668+0000","last_became_peered":"2023-09-07T11:01:22.686668+0000","last_unstale":"2023-09-07T11:01:23.596771+0000","last_undegraded":"2023-09-07T11:01:23.596771+0000","last_fullsized":"2023-09-07T11:01:23.596771+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_clean_scrub_stamp":"2023-09-07T11:01:21.579328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-07T11:01:23.595615+0000","last_change":"2023-09-07T11:01:22.666960+0000","last_active":"2023-09-07T11:01:23.595615+0000","last_peered":"2023-09-07T11:01:23.595615+0000","last_clean":"2023-09-07T11:01:23.595615+0000","last_became_active":"2023-09-07T11:01:22.664499+0000","last_became_peered":"2023-09-07T11:01:22.664499+0000","last_unstale":"2023-09-07T11:01:23.595615+0000","last_undegraded":"2023-09-07T11:01:23.595615+0000","last_fullsized":"2023-09-07T11:01:23.595615+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_clean_scrub_stamp":"2023-09-07T11:01:21.579328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-07T11:01:22.678205+0000","last_change":"2023-09-07T11:01:22.678276+0000","last_active":"2023-09-07T11:01:22.678205+0000","last_peered":"2023-09-07T11:01:22.678205+0000","last_clean":"2023-09-07T11:01:21.579328+0000","last_became_active":"2023-09-07T11:01:22.676344+0000","last_became_peered":"2023-09-07T11:01:22.676344+0000","last_unstale":"2023-09-07T11:01:22.678205+0000","last_undegraded":"2023-09-07T11:01:22.678205+0000","last_fullsized":"2023-09-07T11:01:22.678205+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_clean_scrub_stamp":"2023-09-07T11:01:21.579328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-07T11:01:22.609366+0000","last_change":"2023-09-07T11:01:21.642987+0000","last_active":"2023-09-07T11:01:22.609366+0000","last_peered":"2023-09-07T11:01:22.609366+0000","last_clean":"2023-09-07T11:01:22.609366+0000","last_became_active":"2023-09-07T11:01:21.638242+0000","last_became_peered":"2023-09-07T11:01:21.638242+0000","last_unstale":"2023-09-07T11:01:22.609366+0000","last_undegraded":"2023-09-07T11:01:22.609366+0000","last_fullsized":"2023-09-07T11:01:22.609366+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-07T11:01:20.572656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-07T11:01:20.572656+0000","last_clean_scrub_stamp":"2023-09-07T11:01:20.572656+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26872,"kb_used_data":48,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344968,"statfs":{"total":96636764160,"available":96609247232,"internally_reserved":0,"allocated":49152,"data_stored":14673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-09-07T11:01:27.066 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-07T11:01:27.179 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:27.175+0000 7f98226ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:27.184 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:27.180+0000 7f98226ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:27.361 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:01:27.363 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-09-07T11:01:27.375 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-09-07T11:01:25.559276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80760,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034760,"statfs":{"total":289910292480,"available":289827594240,"internally_reserved":0,"allocated":270336,"data_stored":1224755,"data_compressed":59920,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.957248"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-07T11:01:22.718489+0000","last_change":"2023-09-07T11:01:22.718599+0000","last_active":"2023-09-07T11:01:22.718489+0000","last_peered":"2023-09-07T11:01:22.718489+0000","last_clean":"2023-09-07T11:01:21.579328+0000","last_became_active":"2023-09-07T11:01:22.714692+0000","last_became_peered":"2023-09-07T11:01:22.714692+0000","last_unstale":"2023-09-07T11:01:22.718489+0000","last_undegraded":"2023-09-07T11:01:22.718489+0000","last_fullsized":"2023-09-07T11:01:22.718489+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_clean_scrub_stamp":"2023-09-07T11:01:21.579328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-07T11:01:22.704476+0000","last_change":"2023-09-07T11:01:22.704528+0000","last_active":"2023-09-07T11:01:22.704476+0000","last_peered":"2023-09-07T11:01:22.704476+0000","last_clean":"2023-09-07T11:01:21.579328+0000","last_became_active":"2023-09-07T11:01:22.702754+0000","last_became_peered":"2023-09-07T11:01:22.702754+0000","last_unstale":"2023-09-07T11:01:22.704476+0000","last_undegraded":"2023-09-07T11:01:22.704476+0000","last_fullsized":"2023-09-07T11:01:22.704476+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_clean_scrub_stamp":"2023-09-07T11:01:21.579328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-07T11:01:22.674342+0000","last_change":"2023-09-07T11:01:22.674524+0000","last_active":"2023-09-07T11:01:22.674342+0000","last_peered":"2023-09-07T11:01:22.674342+0000","last_clean":"2023-09-07T11:01:21.579328+0000","last_became_active":"2023-09-07T11:01:22.655450+0000","last_became_peered":"2023-09-07T11:01:22.655450+0000","last_unstale":"2023-09-07T11:01:22.674342+0000","last_undegraded":"2023-09-07T11:01:22.674342+0000","last_fullsized":"2023-09-07T11:01:22.674342+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_clean_scrub_stamp":"2023-09-07T11:01:21.579328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-07T11:01:22.704653+0000","last_change":"2023-09-07T11:01:22.704716+0000","last_active":"2023-09-07T11:01:22.704653+0000","last_peered":"2023-09-07T11:01:22.704653+0000","last_clean":"2023-09-07T11:01:21.579328+0000","last_became_active":"2023-09-07T11:01:22.702322+0000","last_became_peered":"2023-09-07T11:01:22.702322+0000","last_unstale":"2023-09-07T11:01:22.704653+0000","last_undegraded":"2023-09-07T11:01:22.704653+0000","last_fullsized":"2023-09-07T11:01:22.704653+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_clean_scrub_stamp":"2023-09-07T11:01:21.579328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-07T11:01:23.621725+0000","last_change":"2023-09-07T11:01:22.694884+0000","last_active":"2023-09-07T11:01:23.621725+0000","last_peered":"2023-09-07T11:01:23.621725+0000","last_clean":"2023-09-07T11:01:23.621725+0000","last_became_active":"2023-09-07T11:01:22.692860+0000","last_became_peered":"2023-09-07T11:01:22.692860+0000","last_unstale":"2023-09-07T11:01:23.621725+0000","last_undegraded":"2023-09-07T11:01:23.621725+0000","last_fullsized":"2023-09-07T11:01:23.621725+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_clean_scrub_stamp":"2023-09-07T11:01:21.579328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-07T11:01:23.596771+0000","last_change":"2023-09-07T11:01:22.688511+0000","last_active":"2023-09-07T11:01:23.596771+0000","last_peered":"2023-09-07T11:01:23.596771+0000","last_clean":"2023-09-07T11:01:23.596771+0000","last_became_active":"2023-09-07T11:01:22.686668+0000","last_became_peered":"2023-09-07T11:01:22.686668+0000","last_unstale":"2023-09-07T11:01:23.596771+0000","last_undegraded":"2023-09-07T11:01:23.596771+0000","last_fullsized":"2023-09-07T11:01:23.596771+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_clean_scrub_stamp":"2023-09-07T11:01:21.579328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-09-07T11:01:23.595615+0000","last_change":"2023-09-07T11:01:22.666960+0000","last_active":"2023-09-07T11:01:23.595615+0000","last_peered":"2023-09-07T11:01:23.595615+0000","last_clean":"2023-09-07T11:01:23.595615+0000","last_became_active":"2023-09-07T11:01:22.664499+0000","last_became_peered":"2023-09-07T11:01:22.664499+0000","last_unstale":"2023-09-07T11:01:23.595615+0000","last_undegraded":"2023-09-07T11:01:23.595615+0000","last_fullsized":"2023-09-07T11:01:23.595615+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_clean_scrub_stamp":"2023-09-07T11:01:21.579328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-07T11:01:22.678205+0000","last_change":"2023-09-07T11:01:22.678276+0000","last_active":"2023-09-07T11:01:22.678205+0000","last_peered":"2023-09-07T11:01:22.678205+0000","last_clean":"2023-09-07T11:01:21.579328+0000","last_became_active":"2023-09-07T11:01:22.676344+0000","last_became_peered":"2023-09-07T11:01:22.676344+0000","last_unstale":"2023-09-07T11:01:22.678205+0000","last_undegraded":"2023-09-07T11:01:22.678205+0000","last_fullsized":"2023-09-07T11:01:22.678205+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-07T11:01:21.579328+0000","last_clean_scrub_stamp":"2023-09-07T11:01:21.579328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":137,"reported_epoch":11,"state":"active+clean","last_fresh":"2023-09-07T11:01:22.609366+0000","last_change":"2023-09-07T11:01:21.642987+0000","last_active":"2023-09-07T11:01:22.609366+0000","last_peered":"2023-09-07T11:01:22.609366+0000","last_clean":"2023-09-07T11:01:22.609366+0000","last_became_active":"2023-09-07T11:01:21.638242+0000","last_became_peered":"2023-09-07T11:01:21.638242+0000","last_unstale":"2023-09-07T11:01:22.609366+0000","last_undegraded":"2023-09-07T11:01:22.609366+0000","last_fullsized":"2023-09-07T11:01:22.609366+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-07T11:01:20.572656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-07T11:01:20.572656+0000","last_clean_scrub_stamp":"2023-09-07T11:01:20.572656+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26872,"kb_used_data":48,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344968,"statfs":{"total":96636764160,"available":96609247232,"internally_reserved":0,"allocated":49152,"data_stored":14673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-09-07T11:01:27.376 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-07T11:01:27.377 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-09-07T11:01:27.377 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-09-07T11:01:27.377 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-07T11:01:27.490 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:27.488+0000 7f4b0cf9f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:27.492 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:27.491+0000 7f4b0cf9f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:27.715 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-09-07T11:01:27.715 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-09-07T11:01:27.725 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-09-07T11:01:27.725 INFO:teuthology.run_tasks:Running task cbt... 2023-09-07T11:01:27.752 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi031.front.sepia.ceph.com', 'osds': ['ubuntu@smithi031.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi031.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi031.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi031.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'readmode': 'rand', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': False}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2023-09-07T11:01:27.753 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2023-09-07T11:01:27.774 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-09-07T11:01:27.774 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-09-07T11:01:27.826 DEBUG:teuthology.orchestra.run.smithi031:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2023-09-07T11:01:27.884 INFO:teuthology.orchestra.run.smithi031.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2023-09-07T11:01:28.281 INFO:tasks.cbt:Installing collectl 2023-09-07T11:01:28.281 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2023-09-07T11:01:28.386 INFO:teuthology.orchestra.run.smithi031.stderr:--2023-09-07 11:01:28-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2023-09-07T11:01:28.526 INFO:teuthology.orchestra.run.smithi031.stderr:Resolving sourceforge.net (sourceforge.net)... 108.138.85.50, 108.138.85.32, 108.138.85.5, ... 2023-09-07T11:01:28.537 INFO:teuthology.orchestra.run.smithi031.stderr:Connecting to sourceforge.net (sourceforge.net)|108.138.85.50|:443... connected. 2023-09-07T11:01:28.617 INFO:teuthology.orchestra.run.smithi031.stderr:HTTP request sent, awaiting response... 302 Moved Temporarily 2023-09-07T11:01:28.617 INFO:teuthology.orchestra.run.smithi031.stderr:Location: https://sourceforge.net/#!/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download [following] 2023-09-07T11:01:28.617 INFO:teuthology.orchestra.run.smithi031.stderr:--2023-09-07 11:01:28-- https://sourceforge.net/ 2023-09-07T11:01:28.618 INFO:teuthology.orchestra.run.smithi031.stderr:Reusing existing connection to sourceforge.net:443. 2023-09-07T11:01:28.631 INFO:teuthology.orchestra.run.smithi031.stderr:HTTP request sent, awaiting response... 200 OK 2023-09-07T11:01:28.631 INFO:teuthology.orchestra.run.smithi031.stderr:Length: 746 [text/html] 2023-09-07T11:01:28.632 INFO:teuthology.orchestra.run.smithi031.stderr:Saving to: ‘collectl.tar.gz’ 2023-09-07T11:01:28.632 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-09-07T11:01:28.632 INFO:teuthology.orchestra.run.smithi031.stderr: 0K 100% 32.9M=0s 2023-09-07T11:01:28.632 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-09-07T11:01:28.633 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07 11:01:28 (32.9 MB/s) - ‘collectl.tar.gz’ saved [746/746] 2023-09-07T11:01:28.633 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-09-07T11:01:28.665 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-09-07T11:01:28.666 INFO:teuthology.orchestra.run.smithi031.stderr:gzip: stdin: not in gzip format 2023-09-07T11:01:28.666 INFO:teuthology.orchestra.run.smithi031.stderr:tar: Child returned status 1 2023-09-07T11:01:28.666 INFO:teuthology.orchestra.run.smithi031.stderr:tar: Error is not recoverable: exiting now 2023-09-07T11:01:28.667 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-09-07T11:01:28.668 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/task/__init__.py", line 122, in __enter__ self.setup() File "/home/teuthworker/src/github.com_ceph_ceph_f4c5809eee86803e43be24d3362e60639f88c948/qa/tasks/cbt.py", line 229, in setup self.install_dependencies() File "/home/teuthworker/src/github.com_ceph_ceph_f4c5809eee86803e43be24d3362e60639f88c948/qa/tasks/cbt.py", line 84, in install_dependencies self.first_mon.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi031 with status 2: 'sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL' 2023-09-07T11:01:28.781 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=bd86be76e4b24e71bc53b2611ec43511 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/task/__init__.py", line 122, in __enter__ self.setup() File "/home/teuthworker/src/github.com_ceph_ceph_f4c5809eee86803e43be24d3362e60639f88c948/qa/tasks/cbt.py", line 229, in setup self.install_dependencies() File "/home/teuthworker/src/github.com_ceph_ceph_f4c5809eee86803e43be24d3362e60639f88c948/qa/tasks/cbt.py", line 84, in install_dependencies self.first_mon.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi031 with status 2: 'sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL' 2023-09-07T11:01:28.786 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2023-09-07T11:01:28.797 DEBUG:teuthology.orchestra.run.smithi031:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2023-09-07T11:01:28.822 INFO:tasks.cbt_performance:Collecting CBT performance data config 2023-09-07T11:01:28.822 INFO:tasks.cbt_performance:reading cbt preferences 2023-09-07T11:01:28.823 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -name 'json_output.*' 2023-09-07T11:01:28.877 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-07T11:01:28.886 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-09-07T11:01:29.029 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:29.022+0000 7f113dc20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:29.031 INFO:teuthology.orchestra.run.smithi031.stderr:2023-09-07T11:01:29.030+0000 7f113dc20640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-09-07T11:01:29.230 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-07T11:01:29.231 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-07T11:01:29.231 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-09-07T11:01:29.232 INFO:teuthology.orchestra.run:waiting for 300 2023-09-07T11:01:29.297 INFO:tasks.ceph.osd.0:Stopped 2023-09-07T11:01:29.298 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-09-07T11:01:29.298 INFO:teuthology.orchestra.run:waiting for 300 2023-09-07T11:01:29.369 INFO:tasks.ceph.osd.1:Stopped 2023-09-07T11:01:29.370 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-09-07T11:01:29.370 INFO:teuthology.orchestra.run:waiting for 300 2023-09-07T11:01:29.441 INFO:tasks.ceph.osd.2:Stopped 2023-09-07T11:01:29.441 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-07T11:01:29.442 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-09-07T11:01:29.442 INFO:teuthology.orchestra.run:waiting for 300 2023-09-07T11:01:29.467 INFO:tasks.ceph.mgr.x:Stopped 2023-09-07T11:01:29.467 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-07T11:01:29.468 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-09-07T11:01:29.468 INFO:teuthology.orchestra.run:waiting for 300 2023-09-07T11:01:29.518 INFO:tasks.ceph.mon.a:Stopped 2023-09-07T11:01:29.519 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-07T11:01:29.519 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-09-07T11:01:29.590 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi031.front.sepia.ceph.com 2023-09-07T11:01:29.591 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-09-07T11:01:29.856 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi031.front.sepia.ceph.com 2023-09-07T11:01:29.857 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-09-07T11:01:29.994 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi031.front.sepia.ceph.com 2023-09-07T11:01:29.995 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-09-07T11:01:30.056 INFO:tasks.ceph:Archiving mon data... 2023-09-07T11:01:30.057 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2023-09-07_10:45:43-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-ref-crimson-only-distro-crimson-smithi/7390804/data/mon.a.tgz 2023-09-07T11:01:30.058 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2023-09-07T11:01:30.073 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.zVqGofpZuF 2023-09-07T11:01:30.074 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.zVqGofpZuF 2023-09-07T11:01:30.165 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.zVqGofpZuF 2023-09-07T11:01:30.255 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.zVqGofpZuF is 126KB 2023-09-07T11:01:30.308 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.zVqGofpZuF 2023-09-07T11:01:30.327 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-07T11:01:30.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-07T11:01:30.399 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-09-07T11:01:30.399 DEBUG:teuthology.orchestra.run.smithi031:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-07T11:01:30.472 INFO:teuthology.orchestra.run.smithi031.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-07T11:01:30.476 INFO:tasks.ceph:Archiving crash dumps... 2023-09-07T11:01:30.483 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2023-09-07_10:45:43-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-ref-crimson-only-distro-crimson-smithi/7390804/remote/smithi031/crash 2023-09-07T11:01:30.485 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-07T11:01:30.547 INFO:tasks.ceph:Compressing logs... 2023-09-07T11:01:30.548 DEBUG:teuthology.orchestra.run.smithi031:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-09-07T11:01:30.632 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-09-07T11:01:30.633 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28542.log 2023-09-07T11:01:30.633 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29471.log 2023-09-07T11:01:30.633 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5/var/log/ceph/ceph-client.admin.28542.log: --verbose -- /var/log/ceph/ceph-client.admin.29877.log 2023-09-07T11:01:30.647 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28542.log.gz 2023-09-07T11:01:30.647 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.29471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29620.log 2023-09-07T11:01:30.647 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29471.log.gz 2023-09-07T11:01:30.648 INFO:teuthology.orchestra.run.smithi031.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-09-07T11:01:30.648 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.29877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29875.log 2023-09-07T11:01:30.649 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29877.log.gz 2023-09-07T11:01:30.649 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30006.log 2023-09-07T11:01:30.649 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.29620.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29620.log.gz 2023-09-07T11:01:30.650 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.29875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29751.log 2023-09-07T11:01:30.650 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29875.log.gz 2023-09-07T11:01:30.650 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.30006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30090.log 2023-09-07T11:01:30.651 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30006.log.gz 2023-09-07T11:01:30.651 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.29751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29750.log 2023-09-07T11:01:30.651 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29751.log.gz 2023-09-07T11:01:30.652 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28820.log 2023-09-07T11:01:30.652 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.30090.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30090.log.gz 2023-09-07T11:01:30.653 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.29750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28740.log 2023-09-07T11:01:30.653 INFO:teuthology.orchestra.run.smithi031.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29750.log.gz 2023-09-07T11:01:30.653 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.28820.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-09-07T11:01:30.654 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28820.log.gz 2023-09-07T11:01:30.654 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30288.log 2023-09-07T11:01:30.654 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.28740.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28740.log.gz 2023-09-07T11:01:30.659 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29415.log 2023-09-07T11:01:30.660 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.30288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28978.log 2023-09-07T11:01:30.660 INFO:teuthology.orchestra.run.smithi031.stderr: 85.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-09-07T11:01:30.660 INFO:teuthology.orchestra.run.smithi031.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30288.log.gz 2023-09-07T11:01:30.661 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30048.log 2023-09-07T11:01:30.661 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.29415.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29415.log.gz 2023-09-07T11:01:30.662 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.28978.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-09-07T11:01:30.662 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28978.log.gz 2023-09-07T11:01:30.663 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.30048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28780.log 2023-09-07T11:01:30.663 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30048.log.gz 2023-09-07T11:01:30.663 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29008.log 2023-09-07T11:01:30.664 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29669.log 2023-09-07T11:01:30.664 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.28780.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28780.log.gz 2023-09-07T11:01:30.665 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.29008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29575.log 2023-09-07T11:01:30.665 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29008.log.gz 2023-09-07T11:01:30.666 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.29669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28630.log 2023-09-07T11:01:30.666 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29669.log.gz 2023-09-07T11:01:30.666 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-09-07T11:01:30.666 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.29575.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29575.log.gz 2023-09-07T11:01:30.667 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.28630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29749.log 2023-09-07T11:01:30.667 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28630.log.gz 2023-09-07T11:01:30.667 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27156.log 2023-09-07T11:01:30.668 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.29749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29964.log 2023-09-07T11:01:30.668 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29749.log.gz 2023-09-07T11:01:30.669 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.tmp-client.admin.27156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30132.log 2023-09-07T11:01:30.669 INFO:teuthology.orchestra.run.smithi031.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27156.log.gz 2023-09-07T11:01:30.670 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.29964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29837.log 2023-09-07T11:01:30.670 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29964.log.gz 2023-09-07T11:01:30.670 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.30132.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-09-07T11:01:30.672 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30132.log.gz 2023-09-07T11:01:30.673 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.29837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29513.log 2023-09-07T11:01:30.673 INFO:teuthology.orchestra.run.smithi031.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29837.log.gz 2023-09-07T11:01:30.674 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-09-07T11:01:30.674 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.29513.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-09-07T11:01:30.675 INFO:teuthology.orchestra.run.smithi031.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29513.log.gz 2023-09-07T11:01:30.676 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-osd.2.log: 75.8% -- replaced with /var/log/ceph/ceph.log.gz 2023-09-07T11:01:30.676 INFO:teuthology.orchestra.run.smithi031.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-09-07T11:01:30.679 INFO:teuthology.orchestra.run.smithi031.stderr: 90.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-09-07T11:01:30.680 INFO:teuthology.orchestra.run.smithi031.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-09-07T11:01:30.681 INFO:teuthology.orchestra.run.smithi031.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-09-07T11:01:30.681 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-09-07T11:01:30.681 INFO:teuthology.orchestra.run.smithi031.stderr:real 0m0.044s 2023-09-07T11:01:30.682 INFO:teuthology.orchestra.run.smithi031.stderr:user 0m0.081s 2023-09-07T11:01:30.682 INFO:teuthology.orchestra.run.smithi031.stderr:sys 0m0.028s 2023-09-07T11:01:30.682 INFO:tasks.ceph:Archiving logs... 2023-09-07T11:01:30.683 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/log/ceph to /home/teuthworker/archive/matan-2023-09-07_10:45:43-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-ref-crimson-only-distro-crimson-smithi/7390804/remote/smithi031/log 2023-09-07T11:01:30.685 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-07T11:01:30.865 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-07T11:01:30.881 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-07T11:01:30.881 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-07T11:01:30.935 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-07T11:01:30.943 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-07T11:01:30.944 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-07T11:01:30.972 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2023-09-07T11:01:30.974 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-07T11:01:30.974 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2023-09-07T11:01:30.974 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv01.front.sepia.ceph.com 2 6 177 64 -757us[ -757us] +/- 87ms 2023-09-07T11:01:30.975 INFO:teuthology.orchestra.run.smithi031.stdout:^* hv02.front.sepia.ceph.com 3 6 177 64 -2087us[-2084us] +/- 50ms 2023-09-07T11:01:30.975 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 1 +3297us[+3297us] +/- 67ms 2023-09-07T11:01:30.975 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-09-07T11:01:30.976 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-07T11:01:31.009 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-07T11:01:31.010 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-07T11:01:31.062 DEBUG:teuthology.orchestra.run.smithi031:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-07T11:01:31.129 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-07T11:01:31.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-07T11:01:31.162 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-07T11:01:31.175 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-07T11:01:31.192 INFO:teuthology.task.internal:Duration was 413.946385 seconds 2023-09-07T11:01:31.193 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-07T11:01:31.223 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-07T11:01:31.223 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-07T11:01:31.273 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-07T11:01:31.484 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-07T11:01:31.484 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2023-09-07T11:01:31.485 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-07T11:01:31.505 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-07T11:01:31.505 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-07T11:01:31.576 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-07T11:01:31.577 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-07T11:01:31.683 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-07T11:01:31.712 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-07T11:01:31.713 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-07T11:01:31.747 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-07T11:01:31.758 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-07T11:01:31.818 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2023-09-07T11:01:31.844 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-07T11:01:31.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-07T11:01:31.901 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-07T11:01:31.921 INFO:teuthology.task.internal:Transferring archived files... 2023-09-07T11:01:31.926 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2023-09-07_10:45:43-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-ref-crimson-only-distro-crimson-smithi/7390804/remote/smithi031 2023-09-07T11:01:31.927 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-07T11:01:31.978 INFO:teuthology.task.internal:Removing archive directory... 2023-09-07T11:01:31.980 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-07T11:01:32.028 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-07T11:01:32.040 INFO:teuthology.task.internal:Not uploading archives. 2023-09-07T11:01:32.041 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-07T11:01:32.069 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-07T11:01:32.070 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-07T11:01:32.085 INFO:teuthology.orchestra.run.smithi031.stdout: 265217 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 7 11:01 /home/ubuntu/cephtest 2023-09-07T11:01:32.086 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-07T11:01:32.119 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-07T11:01:32.263 INFO:teuthology.nuke:Checking targets against current locks 2023-09-07T11:01:32.292 DEBUG:teuthology.nuke:shortname: smithi031 2023-09-07T11:01:32.293 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-07T11:01:32.318 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-09-07_10:45:43-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-ref-crimson-only-distro-crimson-smithi/7390804', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-07 10:46:58.224355', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGaX5e7rZpui2YElhU102jZW5H2XNqBZ4e0SP9T451dXzh+rj/ICxhb9YL6Yl+7gbPr8wbmGmold07FAy9yrfRk='} 2023-09-07T11:01:32.407 INFO:teuthology.orchestra.console.smithi031:Power off 2023-09-07T11:01:32.408 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-07T11:01:32.445 DEBUG:teuthology.orchestra.console.smithi031:power off output: Chassis Power Control: Down/Off 2023-09-07T11:01:32.446 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-07T11:01:32.488 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-09-07T11:01:36.491 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-07T11:01:36.614 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-09-07T11:01:40.617 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-07T11:01:40.727 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is off 2023-09-07T11:01:40.828 INFO:teuthology.orchestra.console.smithi031:Power off completed 2023-09-07T11:01:40.939 INFO:teuthology.run:Summary data: description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} duration: 413.9463846683502 failure_reason: 'Command failed on smithi031 with status 2: ''sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL''' flavor: crimson owner: scheduled_matan@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=bd86be76e4b24e71bc53b2611ec43511 status: fail success: false 2023-09-07T11:01:40.939 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-07T11:01:41.025 INFO:teuthology.run:FAIL