2023-11-01T00:05:41.956 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-01T00:05:41.957 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi/7442572 --owner scheduled_matan@teuthology --verbose --name matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} -- /home/teuthworker/archive/matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi/7442572/orig.config.yaml 2023-11-01T00:05:42.000 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-01T00:05:42.088 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi/7442572 branch: wip-matanb-crimson-do_osd_ops_execute-part3 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} email: null first_in_suite: false job_id: '7442572' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-matanb-crimson-do_osd_ops_execute-part3 ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) sha1: d1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: crimson sha1: d1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: a9edd23b5a8ddacab225824948b06b370baf276b owner: scheduled_matan@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 6029 sha1: d1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_a9edd23b5a8ddacab225824948b06b370baf276b/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: a9edd23b5a8ddacab225824948b06b370baf276b targets: smithi072.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPqQ8m1lyC4gfmIhCRK9ue+TQpRlEZDrwDAYyVhYmeVhgru8v0m8+bPoyhVsLbnuMQxri2+3S78DUdW7KMv+nhg= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: radosbench: concurrent_ops: 4 concurrent_procs: 2 op_size: - 4194304 pool_monitoring_list: - collectl pool_profile: replicated run_monitoring_list: - collectl time: 60 write_only: true cluster: iterations: 1 osds_per_node: 3 pool_profiles: replicated: pg_size: 256 pgp_size: 256 replication: replicated user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-10-31_22:58:07 tube: smithi user: matan verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.770726 2023-11-01T00:05:42.089 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_a9edd23b5a8ddacab225824948b06b370baf276b/qa; will attempt to use it 2023-11-01T00:05:42.089 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_a9edd23b5a8ddacab225824948b06b370baf276b/qa/tasks 2023-11-01T00:05:42.090 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-01T00:05:42.092 INFO:teuthology.task.internal:Checking packages... 2023-11-01T00:05:42.114 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash 'd1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2' 2023-11-01T00:05:42.114 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-01T00:05:42.115 INFO:teuthology.packaging:ref: None 2023-11-01T00:05:42.115 INFO:teuthology.packaging:tag: None 2023-11-01T00:05:42.116 INFO:teuthology.packaging:branch: wip-matanb-crimson-do_osd_ops_execute-part3 2023-11-01T00:05:42.116 INFO:teuthology.packaging:sha1: d1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2 2023-11-01T00:05:42.116 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-matanb-crimson-do_osd_ops_execute-part3 2023-11-01T00:05:42.276 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6795.gd1dcabba 2023-11-01T00:05:42.276 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-01T00:05:42.283 INFO:teuthology.task.internal:no buildpackages task found 2023-11-01T00:05:42.283 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-01T00:05:42.299 INFO:teuthology.task.internal:Saving configuration 2023-11-01T00:05:42.314 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-01T00:05:42.322 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-01T00:05:42.348 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi072.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi/7442572', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-01 00:00:00.606917', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPqQ8m1lyC4gfmIhCRK9ue+TQpRlEZDrwDAYyVhYmeVhgru8v0m8+bPoyhVsLbnuMQxri2+3S78DUdW7KMv+nhg='} 2023-11-01T00:05:42.349 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-01T00:05:42.359 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-11-01T00:05:42.359 INFO:teuthology.run_tasks:Running task console_log... 2023-11-01T00:05:42.404 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9dbb7b3940>, signals=[15]) 2023-11-01T00:05:42.405 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-01T00:05:42.417 INFO:teuthology.task.internal:Opening connections... 2023-11-01T00:05:42.417 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:05:42.419 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T00:05:42.503 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-01T00:05:42.509 DEBUG:teuthology.orchestra.run.smithi072:> uname -m 2023-11-01T00:05:42.570 INFO:teuthology.orchestra.run.smithi072.stdout:x86_64 2023-11-01T00:05:42.571 DEBUG:teuthology.orchestra.run.smithi072:> cat /etc/os-release 2023-11-01T00:05:42.627 INFO:teuthology.orchestra.run.smithi072.stdout:NAME="CentOS Stream" 2023-11-01T00:05:42.627 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION="9" 2023-11-01T00:05:42.627 INFO:teuthology.orchestra.run.smithi072.stdout:ID="centos" 2023-11-01T00:05:42.627 INFO:teuthology.orchestra.run.smithi072.stdout:ID_LIKE="rhel fedora" 2023-11-01T00:05:42.628 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION_ID="9" 2023-11-01T00:05:42.628 INFO:teuthology.orchestra.run.smithi072.stdout:PLATFORM_ID="platform:el9" 2023-11-01T00:05:42.628 INFO:teuthology.orchestra.run.smithi072.stdout:PRETTY_NAME="CentOS Stream 9" 2023-11-01T00:05:42.628 INFO:teuthology.orchestra.run.smithi072.stdout:ANSI_COLOR="0;31" 2023-11-01T00:05:42.628 INFO:teuthology.orchestra.run.smithi072.stdout:LOGO="fedora-logo-icon" 2023-11-01T00:05:42.628 INFO:teuthology.orchestra.run.smithi072.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-11-01T00:05:42.629 INFO:teuthology.orchestra.run.smithi072.stdout:HOME_URL="https://centos.org/" 2023-11-01T00:05:42.629 INFO:teuthology.orchestra.run.smithi072.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-01T00:05:42.629 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-11-01T00:05:42.629 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-01T00:05:42.630 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server 2023-11-01T00:05:42.654 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-01T00:05:42.668 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-01T00:05:42.729 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-01T00:05:42.730 DEBUG:teuthology.orchestra.run.smithi072:> test '!' -e /home/ubuntu/cephtest 2023-11-01T00:05:42.746 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-01T00:05:42.757 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-01T00:05:42.758 DEBUG:teuthology.orchestra.run.smithi072:> test -z $(ls -A /var/lib/ceph) 2023-11-01T00:05:42.809 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-01T00:05:42.848 INFO:teuthology.run_tasks:Running task kernel... 2023-11-01T00:05:42.868 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-01T00:05:42.868 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-01T00:05:42.869 DEBUG:teuthology.orchestra.run.smithi072:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-01T00:05:42.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T00:05:42.885 DEBUG:teuthology.orchestra.run.smithi072:> uname -r 2023-11-01T00:05:42.941 INFO:teuthology.orchestra.run.smithi072.stdout:5.14.0-361.el9.x86_64 2023-11-01T00:05:42.941 INFO:teuthology.task.kernel:Running kernel on smithi072: 5.14.0-361.el9.x86_64 2023-11-01T00:05:42.942 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum install -y kernel 2023-11-01T00:05:45.986 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - AppStream 13 MB/s | 6.3 MB 00:00 2023-11-01T00:05:47.320 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2023-11-01T00:05:48.962 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - BaseOS 6.5 MB/s | 7.9 MB 00:01 2023-11-01T00:05:50.816 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - AppStream 22 MB/s | 18 MB 00:00 2023-11-01T00:05:56.234 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 5.8 MB 00:02 2023-11-01T00:05:57.553 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - Extras packages 68 kB/s | 15 kB 00:00 2023-11-01T00:06:05.943 INFO:teuthology.orchestra.run.smithi072.stdout:Extra Packages for Enterprise Linux 2.3 MB/s | 19 MB 00:08 2023-11-01T00:06:10.424 INFO:teuthology.orchestra.run.smithi072.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2023-11-01T00:06:12.523 INFO:teuthology.orchestra.run.smithi072.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-11-01T00:06:12.523 INFO:teuthology.orchestra.run.smithi072.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-11-01T00:06:12.622 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:06:12.637 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:06:12.638 INFO:teuthology.orchestra.run.smithi072.stdout: Package Architecture Version Repository Size 2023-11-01T00:06:12.638 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:06:12.638 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-11-01T00:06:12.638 INFO:teuthology.orchestra.run.smithi072.stdout: kernel x86_64 5.14.0-378.el9 baseos 5.3 M 2023-11-01T00:06:12.638 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2023-11-01T00:06:12.639 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-core x86_64 5.14.0-378.el9 baseos 20 M 2023-11-01T00:06:12.639 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules x86_64 5.14.0-378.el9 baseos 38 M 2023-11-01T00:06:12.639 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-core x86_64 5.14.0-378.el9 baseos 32 M 2023-11-01T00:06:12.639 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:06:12.639 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-11-01T00:06:12.640 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:06:12.640 INFO:teuthology.orchestra.run.smithi072.stdout:Install 4 Packages 2023-11-01T00:06:12.640 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:06:12.641 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 95 M 2023-11-01T00:06:12.641 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 120 M 2023-11-01T00:06:12.641 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-11-01T00:06:13.718 INFO:teuthology.orchestra.run.smithi072.stdout:(1/4): kernel-5.14.0-378.el9.x86_64.rpm 5.2 MB/s | 5.3 MB 00:01 2023-11-01T00:06:14.594 INFO:teuthology.orchestra.run.smithi072.stdout:(2/4): kernel-core-5.14.0-378.el9.x86_64.rpm 11 MB/s | 20 MB 00:01 2023-11-01T00:06:15.636 INFO:teuthology.orchestra.run.smithi072.stdout:(3/4): kernel-modules-core-5.14.0-378.el9.x86_6 17 MB/s | 32 MB 00:01 2023-11-01T00:06:16.261 INFO:teuthology.orchestra.run.smithi072.stdout:(4/4): kernel-modules-5.14.0-378.el9.x86_64.rpm 11 MB/s | 38 MB 00:03 2023-11-01T00:06:16.262 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-11-01T00:06:16.262 INFO:teuthology.orchestra.run.smithi072.stdout:Total 26 MB/s | 95 MB 00:03 2023-11-01T00:06:17.099 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-11-01T00:06:17.488 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-11-01T00:06:17.488 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-11-01T00:06:18.789 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-11-01T00:06:18.789 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-11-01T00:06:21.457 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-11-01T00:06:22.164 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-modules-core-5.14.0-378.el9.x86_64 1/4 2023-11-01T00:06:22.342 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-11-01T00:06:25.185 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 2/4 2023-11-01T00:06:25.444 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-11-01T00:06:30.055 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-11-01T00:06:30.196 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-5.14.0-378.el9.x86_64 4/4 2023-11-01T00:06:34.813 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-11-01T00:07:05.614 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-core-5.14.0-378.el9.x86_64 4/4 2023-11-01T00:07:05.656 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-5.14.0-378.el9.x86_64 4/4 2023-11-01T00:07:06.053 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-5.14.0-378.el9.x86_64 4/4 2023-11-01T00:07:06.053 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-5.14.0-378.el9.x86_64 1/4 2023-11-01T00:07:06.053 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-core-5.14.0-378.el9.x86_64 2/4 2023-11-01T00:07:06.054 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-modules-5.14.0-378.el9.x86_64 3/4 2023-11-01T00:07:06.453 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-modules-core-5.14.0-378.el9.x86_64 4/4 2023-11-01T00:07:06.453 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:07:06.453 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-11-01T00:07:06.454 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-5.14.0-378.el9.x86_64 2023-11-01T00:07:06.454 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-core-5.14.0-378.el9.x86_64 2023-11-01T00:07:06.454 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-5.14.0-378.el9.x86_64 2023-11-01T00:07:06.454 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-core-5.14.0-378.el9.x86_64 2023-11-01T00:07:06.454 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:07:06.455 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:07:07.147 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q kernel | sort -rV | head -n 1 2023-11-01T00:07:07.318 INFO:teuthology.orchestra.run.smithi072.stdout:kernel-5.14.0-378.el9.x86_64 2023-11-01T00:07:07.319 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-11-01T00:07:07.319 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-378.el9.x86_64 2023-11-01T00:07:07.319 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-01T00:07:07.320 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-01T00:07:07.320 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-11-01T00:07:07.320 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi072.front.sepia.ceph.com, path=None, version=distro) 2023-11-01T00:07:07.320 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q kernel | sort -rV | head -n 1 2023-11-01T00:07:07.437 INFO:teuthology.orchestra.run.smithi072.stdout:kernel-5.14.0-378.el9.x86_64 2023-11-01T00:07:07.437 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-378.el9.x86_64 2023-11-01T00:07:07.438 DEBUG:teuthology.orchestra.run.smithi072:> sudo rpm -qi grub2-tools 2023-11-01T00:07:07.474 INFO:teuthology.orchestra.run.smithi072.stdout:Name : grub2-tools 2023-11-01T00:07:07.475 INFO:teuthology.orchestra.run.smithi072.stdout:Epoch : 1 2023-11-01T00:07:07.475 INFO:teuthology.orchestra.run.smithi072.stdout:Version : 2.06 2023-11-01T00:07:07.475 INFO:teuthology.orchestra.run.smithi072.stdout:Release : 68.el9 2023-11-01T00:07:07.475 INFO:teuthology.orchestra.run.smithi072.stdout:Architecture: x86_64 2023-11-01T00:07:07.476 INFO:teuthology.orchestra.run.smithi072.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-11-01T00:07:07.476 INFO:teuthology.orchestra.run.smithi072.stdout:Group : Unspecified 2023-11-01T00:07:07.476 INFO:teuthology.orchestra.run.smithi072.stdout:Size : 8264254 2023-11-01T00:07:07.476 INFO:teuthology.orchestra.run.smithi072.stdout:License : GPLv3+ 2023-11-01T00:07:07.476 INFO:teuthology.orchestra.run.smithi072.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-11-01T00:07:07.477 INFO:teuthology.orchestra.run.smithi072.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-11-01T00:07:07.477 INFO:teuthology.orchestra.run.smithi072.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-11-01T00:07:07.477 INFO:teuthology.orchestra.run.smithi072.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-11-01T00:07:07.477 INFO:teuthology.orchestra.run.smithi072.stdout:Packager : builder@centos.org 2023-11-01T00:07:07.478 INFO:teuthology.orchestra.run.smithi072.stdout:Vendor : CentOS 2023-11-01T00:07:07.478 INFO:teuthology.orchestra.run.smithi072.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-01T00:07:07.478 INFO:teuthology.orchestra.run.smithi072.stdout:Summary : Support tools for GRUB. 2023-11-01T00:07:07.478 INFO:teuthology.orchestra.run.smithi072.stdout:Description : 2023-11-01T00:07:07.479 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:07:07.479 INFO:teuthology.orchestra.run.smithi072.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-01T00:07:07.479 INFO:teuthology.orchestra.run.smithi072.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-01T00:07:07.479 INFO:teuthology.orchestra.run.smithi072.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-01T00:07:07.480 INFO:teuthology.orchestra.run.smithi072.stdout:hardware devices. 2023-11-01T00:07:07.480 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:07:07.480 INFO:teuthology.orchestra.run.smithi072.stdout:This subpackage provides tools for support of all platforms. 2023-11-01T00:07:07.481 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-01T00:07:07.481 INFO:teuthology.task.kernel:Updating grub on smithi072 to boot 5.14.0-378.el9.x86_64 2023-11-01T00:07:07.482 DEBUG:teuthology.orchestra.run.smithi072:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-01T00:07:08.094 INFO:teuthology.orchestra.run.smithi072.stderr:Generating grub configuration file ... 2023-11-01T00:07:09.680 INFO:teuthology.orchestra.run.smithi072.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-11-01T00:07:09.703 INFO:teuthology.orchestra.run.smithi072.stderr:done 2023-11-01T00:07:09.705 DEBUG:teuthology.orchestra.run.smithi072:> mktemp 2023-11-01T00:07:09.725 INFO:teuthology.orchestra.run.smithi072.stdout:/tmp/tmp.q8ayWni9jC 2023-11-01T00:07:09.726 DEBUG:teuthology.orchestra.run.smithi072:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.q8ayWni9jC 2023-11-01T00:07:09.793 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0666 /tmp/tmp.q8ayWni9jC 2023-11-01T00:07:09.937 DEBUG:teuthology.orchestra.remote:smithi072:/tmp/tmp.q8ayWni9jC is 6KB 2023-11-01T00:07:09.956 DEBUG:teuthology.orchestra.run.smithi072:> rm -fr /tmp/tmp.q8ayWni9jC 2023-11-01T00:07:09.970 DEBUG:teuthology.orchestra.run.smithi072:> sudo /bin/ls /boot/loader/entries || true 2023-11-01T00:07:10.035 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-11-01T00:07:10.035 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-11-01T00:07:10.035 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-11-01T00:07:10.036 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64.conf 2023-11-01T00:07:10.037 DEBUG:teuthology.orchestra.run.smithi072:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-378.el9.x86_64 2023-11-01T00:07:10.181 DEBUG:teuthology.orchestra.run.smithi072:> sudo shutdown -r now 2023-11-01T00:07:40.214 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-11-01T00:07:40.215 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-11-01T00:07:40.216 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T00:07:58.586 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.72 2023-11-01T00:08:07.594 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-11-01T00:08:07.595 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T00:08:10.650 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.72 2023-11-01T00:08:22.662 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-11-01T00:08:22.663 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T00:08:41.082 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.72 2023-11-01T00:08:56.094 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-11-01T00:08:56.095 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T00:08:59.830 DEBUG:teuthology.orchestra.run.smithi072:> true 2023-11-01T00:09:00.911 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-11-01T00:09:00.912 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-378.el9.x86_64"... 2023-11-01T00:09:00.912 DEBUG:teuthology.orchestra.run.smithi072:> uname -r 2023-11-01T00:09:00.926 INFO:teuthology.orchestra.run.smithi072.stdout:5.14.0-378.el9.x86_64 2023-11-01T00:09:00.927 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-378.el9.x86_64 vs 5.14.0-378.el9.x86_64 2023-11-01T00:09:00.927 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-01T00:09:00.927 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-01T00:09:01.928 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-11-01T00:09:01.929 DEBUG:teuthology.orchestra.run.smithi072:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-01T00:09:02.067 INFO:teuthology.orchestra.run.smithi072.stdout:ttyS1 2023-11-01T00:09:02.106 DEBUG:teuthology.parallel:result is None 2023-11-01T00:09:02.107 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-01T00:09:02.114 INFO:teuthology.task.internal:Creating test directory... 2023-11-01T00:09:02.114 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-01T00:09:02.129 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-01T00:09:02.135 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-01T00:09:02.145 INFO:teuthology.task.internal:Creating archive directory... 2023-11-01T00:09:02.146 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-01T00:09:02.210 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-01T00:09:02.216 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-01T00:09:02.217 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-01T00:09:02.363 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-01T00:09:02.376 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-01T00:09:02.378 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-01T00:09:02.385 INFO:teuthology.task.internal:Configuring sudo... 2023-11-01T00:09:02.386 DEBUG:teuthology.orchestra.run.smithi072:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-01T00:09:02.412 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-01T00:09:02.420 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-01T00:09:02.421 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-01T00:09:02.468 DEBUG:teuthology.orchestra.run.smithi072:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-01T00:09:02.552 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-01T00:09:02.600 DEBUG:teuthology.orchestra.run.smithi072:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-01T00:09:02.685 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-01T00:09:02.728 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-11-01T00:09:02.728 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-01T00:09:02.795 DEBUG:teuthology.orchestra.run.smithi072:> sudo service rsyslog restart 2023-11-01T00:09:02.877 INFO:teuthology.orchestra.run.smithi072.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-01T00:09:03.232 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-01T00:09:03.243 INFO:teuthology.task.internal:Starting timer... 2023-11-01T00:09:03.244 INFO:teuthology.run_tasks:Running task pcp... 2023-11-01T00:09:03.259 INFO:teuthology.run_tasks:Running task selinux... 2023-11-01T00:09:03.272 DEBUG:teuthology.orchestra.run.smithi072:> sudo service auditd rotate 2023-11-01T00:09:03.379 INFO:teuthology.orchestra.run.smithi072.stdout:Rotating logs: 2023-11-01T00:09:03.381 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-01T00:09:03.381 DEBUG:teuthology.orchestra.run.smithi072:> /usr/sbin/getenforce 2023-11-01T00:09:03.407 INFO:teuthology.orchestra.run.smithi072.stdout:Permissive 2023-11-01T00:09:03.407 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi072.front.sepia.ceph.com': 'permissive'} 2023-11-01T00:09:03.408 DEBUG:teuthology.orchestra.run.smithi072:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-01T00:09:03.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T00:09:03.479 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-01T00:09:03.479 DEBUG:teuthology.orchestra.run.smithi072:> sudo /usr/sbin/setenforce permissive 2023-11-01T00:09:03.559 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-01T00:09:03.573 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-01T00:09:03.574 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-01T00:09:03.590 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-01T00:09:03.592 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi072.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-01T00:14:28.244 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi072.front.sepia.ceph.com')] 2023-11-01T00:14:28.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-11-01T00:14:28.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-01T00:14:28.332 DEBUG:teuthology.orchestra.run.smithi072:> true 2023-11-01T00:14:28.401 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-11-01T00:14:28.402 INFO:teuthology.run_tasks:Running task clock... 2023-11-01T00:14:28.415 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-01T00:14:28.416 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-01T00:14:28.417 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-01T00:14:28.471 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-01T00:14:28.488 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-01T00:14:28.517 INFO:teuthology.orchestra.run.smithi072.stderr:sudo: ntpd: command not found 2023-11-01T00:14:28.530 INFO:teuthology.orchestra.run.smithi072.stdout:506 Cannot talk to daemon 2023-11-01T00:14:28.547 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-01T00:14:28.563 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-01T00:14:28.606 INFO:teuthology.orchestra.run.smithi072.stderr:bash: line 1: ntpq: command not found 2023-11-01T00:14:28.610 INFO:teuthology.orchestra.run.smithi072.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-01T00:14:28.610 INFO:teuthology.orchestra.run.smithi072.stdout:=============================================================================== 2023-11-01T00:14:28.610 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-01T00:14:28.610 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-01T00:14:28.610 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-01T00:14:28.611 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-01T00:14:28.612 INFO:teuthology.run_tasks:Running task install... 2023-11-01T00:14:28.621 DEBUG:teuthology.task.install:project ceph 2023-11-01T00:14:28.622 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': 'd1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2'}} 2023-11-01T00:14:28.622 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': 'd1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2'} 2023-11-01T00:14:28.622 INFO:teuthology.task.install:Using flavor: crimson 2023-11-01T00:14:28.628 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-01T00:14:28.628 INFO:teuthology.task.install:extra packages: [] 2023-11-01T00:14:28.628 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2', 'tag': None, 'wait_for_package': False} 2023-11-01T00:14:28.629 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=d1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2 2023-11-01T00:14:28.775 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-do_osd_ops_execute-part3/d1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2/centos/9/flavors/crimson/ 2023-11-01T00:14:28.775 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6795.gd1dcabba 2023-11-01T00:14:28.883 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-do_osd_ops_execute-part3/d1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-do_osd_ops_execute-part3/d1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-matanb-crimson-do_osd_ops_execute-part3/d1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-01T00:14:28.884 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-11-01T00:14:28.884 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-01T00:14:28.912 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-01T00:14:28.913 DEBUG:teuthology.orchestra.run.smithi072:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-01T00:14:28.981 DEBUG:teuthology.orchestra.run.smithi072:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-01T00:14:29.060 DEBUG:teuthology.orchestra.run.smithi072:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-01T00:14:29.126 INFO:teuthology.orchestra.run.smithi072.stdout:check_obsoletes = 1 2023-11-01T00:14:29.128 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum clean all 2023-11-01T00:14:29.362 INFO:teuthology.orchestra.run.smithi072.stdout:72 files removed 2023-11-01T00:14:29.389 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-radosgw 2023-11-01T00:14:30.016 INFO:teuthology.orchestra.run.smithi072.stdout:ceph packages for x86_64 259 kB/s | 81 kB 00:00 2023-11-01T00:14:30.375 INFO:teuthology.orchestra.run.smithi072.stdout:ceph noarch packages 30 kB/s | 9.5 kB 00:00 2023-11-01T00:14:30.685 INFO:teuthology.orchestra.run.smithi072.stdout:ceph source packages 7.2 kB/s | 1.9 kB 00:00 2023-11-01T00:14:31.089 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - AppStream 17 MB/s | 6.3 MB 00:00 2023-11-01T00:14:32.407 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2023-11-01T00:14:33.441 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 7.9 MB 00:00 2023-11-01T00:14:36.220 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - AppStream 10 MB/s | 18 MB 00:01 2023-11-01T00:14:41.864 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 5.8 MB 00:02 2023-11-01T00:14:43.282 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 15 kB 00:00 2023-11-01T00:14:45.338 INFO:teuthology.orchestra.run.smithi072.stdout:Extra Packages for Enterprise Linux 9.9 MB/s | 19 MB 00:01 2023-11-01T00:14:49.779 INFO:teuthology.orchestra.run.smithi072.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2023-11-01T00:14:52.197 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:14:52.215 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:14:52.216 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2023-11-01T00:14:52.216 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:14:52.216 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-11-01T00:14:52.216 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-radosgw x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 8.4 M 2023-11-01T00:14:52.216 INFO:teuthology.orchestra.run.smithi072.stdout:Upgrading: 2023-11-01T00:14:52.217 INFO:teuthology.orchestra.run.smithi072.stdout: librados2 x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 3.1 M 2023-11-01T00:14:52.217 INFO:teuthology.orchestra.run.smithi072.stdout: librbd1 x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 3.0 M 2023-11-01T00:14:52.217 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2023-11-01T00:14:52.217 INFO:teuthology.orchestra.run.smithi072.stdout: boost-program-options 2023-11-01T00:14:52.217 INFO:teuthology.orchestra.run.smithi072.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-11-01T00:14:52.218 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-base x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 6.6 M 2023-11-01T00:14:52.218 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-common x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 17 M 2023-11-01T00:14:52.218 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-selinux x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 24 k 2023-11-01T00:14:52.218 INFO:teuthology.orchestra.run.smithi072.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-11-01T00:14:52.218 INFO:teuthology.orchestra.run.smithi072.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-11-01T00:14:52.219 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-11-01T00:14:52.219 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-11-01T00:14:52.219 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs2 x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 727 k 2023-11-01T00:14:52.219 INFO:teuthology.orchestra.run.smithi072.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-11-01T00:14:52.219 INFO:teuthology.orchestra.run.smithi072.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-11-01T00:14:52.220 INFO:teuthology.orchestra.run.smithi072.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-11-01T00:14:52.220 INFO:teuthology.orchestra.run.smithi072.stdout: libradosstriper1 x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 469 k 2023-11-01T00:14:52.220 INFO:teuthology.orchestra.run.smithi072.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-11-01T00:14:52.220 INFO:teuthology.orchestra.run.smithi072.stdout: librgw2 x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 4.5 M 2023-11-01T00:14:52.220 INFO:teuthology.orchestra.run.smithi072.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-11-01T00:14:52.220 INFO:teuthology.orchestra.run.smithi072.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-11-01T00:14:52.221 INFO:teuthology.orchestra.run.smithi072.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-11-01T00:14:52.221 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-argparse 2023-11-01T00:14:52.221 INFO:teuthology.orchestra.run.smithi072.stdout: x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 45 k 2023-11-01T00:14:52.221 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-common x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 127 k 2023-11-01T00:14:52.221 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cephfs x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 162 k 2023-11-01T00:14:52.222 INFO:teuthology.orchestra.run.smithi072.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-11-01T00:14:52.222 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rados x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 321 k 2023-11-01T00:14:52.222 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rbd x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 298 k 2023-11-01T00:14:52.222 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rgw x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 99 k 2023-11-01T00:14:52.222 INFO:teuthology.orchestra.run.smithi072.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-11-01T00:14:52.223 INFO:teuthology.orchestra.run.smithi072.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-11-01T00:14:52.223 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:14:52.223 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-11-01T00:14:52.223 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:14:52.223 INFO:teuthology.orchestra.run.smithi072.stdout:Install 28 Packages 2023-11-01T00:14:52.224 INFO:teuthology.orchestra.run.smithi072.stdout:Upgrade 2 Packages 2023-11-01T00:14:52.224 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:14:52.224 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 54 M 2023-11-01T00:14:52.224 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-11-01T00:14:55.398 INFO:teuthology.orchestra.run.smithi072.stdout:(1/30): ceph-base-18.0.0-6795.gd1dcabba.el9.x86 2.3 MB/s | 6.6 MB 00:02 2023-11-01T00:14:55.549 INFO:teuthology.orchestra.run.smithi072.stdout:(2/30): ceph-radosgw-18.0.0-6795.gd1dcabba.el9. 2.8 MB/s | 8.4 MB 00:03 2023-11-01T00:14:55.566 INFO:teuthology.orchestra.run.smithi072.stdout:(3/30): ceph-selinux-18.0.0-6795.gd1dcabba.el9. 144 kB/s | 24 kB 00:00 2023-11-01T00:14:55.666 INFO:teuthology.orchestra.run.smithi072.stdout:(4/30): libradosstriper1-18.0.0-6795.gd1dcabba. 4.6 MB/s | 469 kB 00:00 2023-11-01T00:14:55.700 INFO:teuthology.orchestra.run.smithi072.stdout:(5/30): libcephfs2-18.0.0-6795.gd1dcabba.el9.x8 4.7 MB/s | 727 kB 00:00 2023-11-01T00:14:55.984 INFO:teuthology.orchestra.run.smithi072.stdout:(6/30): ceph-common-18.0.0-6795.gd1dcabba.el9.x 5.0 MB/s | 17 MB 00:03 2023-11-01T00:14:56.001 INFO:teuthology.orchestra.run.smithi072.stdout:(7/30): python3-ceph-argparse-18.0.0-6795.gd1dc 150 kB/s | 45 kB 00:00 2023-11-01T00:14:56.035 INFO:teuthology.orchestra.run.smithi072.stdout:(8/30): python3-ceph-common-18.0.0-6795.gd1dcab 2.5 MB/s | 127 kB 00:00 2023-11-01T00:14:56.079 INFO:teuthology.orchestra.run.smithi072.stdout:(9/30): python3-cephfs-18.0.0-6795.gd1dcabba.el 2.0 MB/s | 162 kB 00:00 2023-11-01T00:14:56.129 INFO:teuthology.orchestra.run.smithi072.stdout:(10/30): python3-rados-18.0.0-6795.gd1dcabba.el 3.3 MB/s | 321 kB 00:00 2023-11-01T00:14:56.255 INFO:teuthology.orchestra.run.smithi072.stdout:(11/30): librgw2-18.0.0-6795.gd1dcabba.el9.x86_ 7.6 MB/s | 4.5 MB 00:00 2023-11-01T00:14:56.280 INFO:teuthology.orchestra.run.smithi072.stdout:(12/30): python3-rgw-18.0.0-6795.gd1dcabba.el9. 658 kB/s | 99 kB 00:00 2023-11-01T00:14:56.306 INFO:teuthology.orchestra.run.smithi072.stdout:(13/30): python3-rbd-18.0.0-6795.gd1dcabba.el9. 1.3 MB/s | 298 kB 00:00 2023-11-01T00:14:56.423 INFO:teuthology.orchestra.run.smithi072.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 631 kB/s | 106 kB 00:00 2023-11-01T00:14:56.457 INFO:teuthology.orchestra.run.smithi072.stdout:(15/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 904 kB/s | 159 kB 00:00 2023-11-01T00:14:56.491 INFO:teuthology.orchestra.run.smithi072.stdout:(16/30): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2023-11-01T00:14:56.533 INFO:teuthology.orchestra.run.smithi072.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 3.6 MB/s | 661 kB 00:00 2023-11-01T00:14:56.567 INFO:teuthology.orchestra.run.smithi072.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 282 kB 00:00 2023-11-01T00:14:56.651 INFO:teuthology.orchestra.run.smithi072.stdout:(19/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 285 kB/s | 45 kB 00:00 2023-11-01T00:14:56.760 INFO:teuthology.orchestra.run.smithi072.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 489 kB/s | 111 kB 00:00 2023-11-01T00:14:56.843 INFO:teuthology.orchestra.run.smithi072.stdout:(21/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2023-11-01T00:14:56.861 INFO:teuthology.orchestra.run.smithi072.stdout:(22/30): libarrow-doc-9.0.0-9.el9.noarch.rpm 243 kB/s | 24 kB 00:00 2023-11-01T00:14:57.061 INFO:teuthology.orchestra.run.smithi072.stdout:(23/30): libarrow-9.0.0-9.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2023-11-01T00:14:57.087 INFO:teuthology.orchestra.run.smithi072.stdout:(24/30): liboath-2.6.7-2.el9.x86_64.rpm 200 kB/s | 49 kB 00:00 2023-11-01T00:14:57.112 INFO:teuthology.orchestra.run.smithi072.stdout:(25/30): libunwind-1.6.2-1.el9.x86_64.rpm 269 kB/s | 67 kB 00:00 2023-11-01T00:14:57.147 INFO:teuthology.orchestra.run.smithi072.stdout:(26/30): parquet-libs-9.0.0-9.el9.x86_64.rpm 9.5 MB/s | 837 kB 00:00 2023-11-01T00:14:57.181 INFO:teuthology.orchestra.run.smithi072.stdout:(27/30): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2023-11-01T00:14:57.273 INFO:teuthology.orchestra.run.smithi072.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 9.9 MB/s | 1.6 MB 00:00 2023-11-01T00:14:57.499 INFO:teuthology.orchestra.run.smithi072.stdout:(29/30): librbd1-18.0.0-6795.gd1dcabba.el9.x86_ 9.6 MB/s | 3.0 MB 00:00 2023-11-01T00:14:57.758 INFO:teuthology.orchestra.run.smithi072.stdout:(30/30): librados2-18.0.0-6795.gd1dcabba.el9.x8 5.1 MB/s | 3.1 MB 00:00 2023-11-01T00:14:57.764 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-11-01T00:14:57.764 INFO:teuthology.orchestra.run.smithi072.stdout:Total 9.7 MB/s | 54 MB 00:05 2023-11-01T00:14:57.776 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-11-01T00:14:57.812 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-11-01T00:14:57.812 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-11-01T00:14:58.277 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-11-01T00:14:58.278 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-11-01T00:14:59.848 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-11-01T00:15:00.257 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2023-11-01T00:15:00.337 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : librados2-2:18.0.0-6795.gd1dcabba.el9.x86_64 2/32 2023-11-01T00:15:00.420 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados2-2:18.0.0-6795.gd1dcabba.el9.x86_64 2/32 2023-11-01T00:15:00.519 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rados-2:18.0.0-6795.gd1dcabba.el9.x86_64 3/32 2023-11-01T00:15:00.550 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephfs2-2:18.0.0-6795.gd1dcabba.el9.x86_64 4/32 2023-11-01T00:15:00.613 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephfs2-2:18.0.0-6795.gd1dcabba.el9.x86_64 4/32 2023-11-01T00:15:00.668 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/32 2023-11-01T00:15:00.697 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/32 2023-11-01T00:15:00.744 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/32 2023-11-01T00:15:00.788 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ceph-argparse-2:18.0.0-6795.gd1dcabba.el9. 8/32 2023-11-01T00:15:00.855 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cephfs-2:18.0.0-6795.gd1dcabba.el9.x86_64 9/32 2023-11-01T00:15:00.881 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libradosstriper1-2:18.0.0-6795.gd1dcabba.el9.x86_6 10/32 2023-11-01T00:15:00.954 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6795.gd1dcabba.el9.x86_6 10/32 2023-11-01T00:15:00.982 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/32 2023-11-01T00:15:01.088 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2023-11-01T00:15:01.126 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/32 2023-11-01T00:15:01.166 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2023-11-01T00:15:01.200 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2023-11-01T00:15:01.236 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/32 2023-11-01T00:15:01.274 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2023-11-01T00:15:01.315 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-11-01T00:15:01.626 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2023-11-01T00:15:01.663 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : librbd1-2:18.0.0-6795.gd1dcabba.el9.x86_64 20/32 2023-11-01T00:15:01.736 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librbd1-2:18.0.0-6795.gd1dcabba.el9.x86_64 20/32 2023-11-01T00:15:01.769 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rbd-2:18.0.0-6795.gd1dcabba.el9.x86_64 21/32 2023-11-01T00:15:01.810 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2023-11-01T00:15:02.238 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 23/32 2023-11-01T00:15:02.273 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librgw2-2:18.0.0-6795.gd1dcabba.el9.x86_64 24/32 2023-11-01T00:15:02.345 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librgw2-2:18.0.0-6795.gd1dcabba.el9.x86_64 24/32 2023-11-01T00:15:02.424 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rgw-2:18.0.0-6795.gd1dcabba.el9.x86_64 25/32 2023-11-01T00:15:02.454 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ceph-common-2:18.0.0-6795.gd1dcabba.el9.x8 26/32 2023-11-01T00:15:04.493 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-common-2:18.0.0-6795.gd1dcabba.el9.x86_64 27/32 2023-11-01T00:15:04.533 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-common-2:18.0.0-6795.gd1dcabba.el9.x86_64 27/32 2023-11-01T00:15:05.275 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-common-2:18.0.0-6795.gd1dcabba.el9.x86_64 27/32 2023-11-01T00:15:05.324 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-base-2:18.0.0-6795.gd1dcabba.el9.x86_64 28/32 2023-11-01T00:15:05.431 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-base-2:18.0.0-6795.gd1dcabba.el9.x86_64 28/32 2023-11-01T00:15:05.432 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-11-01T00:15:05.432 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-11-01T00:15:05.432 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:15:05.685 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-selinux-2:18.0.0-6795.gd1dcabba.el9.x86_64 29/32 2023-11-01T00:15:16.788 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6795.gd1dcabba.el9.x86_64 29/32 2023-11-01T00:15:16.788 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /sys 2023-11-01T00:15:16.788 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /proc 2023-11-01T00:15:16.789 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /mnt 2023-11-01T00:15:16.789 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /var/tmp 2023-11-01T00:15:16.789 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /home 2023-11-01T00:15:16.789 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /root 2023-11-01T00:15:16.789 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /tmp 2023-11-01T00:15:16.790 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:15:17.664 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-radosgw-2:18.0.0-6795.gd1dcabba.el9.x86_64 30/32 2023-11-01T00:15:17.695 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6795.gd1dcabba.el9.x86_64 30/32 2023-11-01T00:15:17.695 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-01T00:15:17.696 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-11-01T00:15:17.696 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-01T00:15:17.696 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-11-01T00:15:17.696 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:15:17.697 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-11-01T00:15:17.777 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2023-11-01T00:15:17.777 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2023-11-01T00:15:19.191 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2023-11-01T00:15:19.192 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-base-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/32 2023-11-01T00:15:19.192 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-common-2:18.0.0-6795.gd1dcabba.el9.x86_64 2/32 2023-11-01T00:15:19.192 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-radosgw-2:18.0.0-6795.gd1dcabba.el9.x86_64 3/32 2023-11-01T00:15:19.192 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-selinux-2:18.0.0-6795.gd1dcabba.el9.x86_64 4/32 2023-11-01T00:15:19.192 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephfs2-2:18.0.0-6795.gd1dcabba.el9.x86_64 5/32 2023-11-01T00:15:19.193 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libradosstriper1-2:18.0.0-6795.gd1dcabba.el9.x86_6 6/32 2023-11-01T00:15:19.193 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librgw2-2:18.0.0-6795.gd1dcabba.el9.x86_64 7/32 2023-11-01T00:15:19.193 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6795.gd1dcabba.el9. 8/32 2023-11-01T00:15:19.193 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ceph-common-2:18.0.0-6795.gd1dcabba.el9.x8 9/32 2023-11-01T00:15:19.193 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cephfs-2:18.0.0-6795.gd1dcabba.el9.x86_64 10/32 2023-11-01T00:15:19.193 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rados-2:18.0.0-6795.gd1dcabba.el9.x86_64 11/32 2023-11-01T00:15:19.194 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rbd-2:18.0.0-6795.gd1dcabba.el9.x86_64 12/32 2023-11-01T00:15:19.194 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rgw-2:18.0.0-6795.gd1dcabba.el9.x86_64 13/32 2023-11-01T00:15:19.194 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2023-11-01T00:15:19.194 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2023-11-01T00:15:19.194 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/32 2023-11-01T00:15:19.195 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/32 2023-11-01T00:15:19.195 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/32 2023-11-01T00:15:19.195 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 19/32 2023-11-01T00:15:19.195 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2023-11-01T00:15:19.195 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2023-11-01T00:15:19.196 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 22/32 2023-11-01T00:15:19.196 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 23/32 2023-11-01T00:15:19.196 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2023-11-01T00:15:19.196 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2023-11-01T00:15:19.196 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 26/32 2023-11-01T00:15:19.197 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2023-11-01T00:15:19.197 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2023-11-01T00:15:19.197 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados2-2:18.0.0-6795.gd1dcabba.el9.x86_64 29/32 2023-11-01T00:15:19.197 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2023-11-01T00:15:19.197 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librbd1-2:18.0.0-6795.gd1dcabba.el9.x86_64 31/32 2023-11-01T00:15:20.169 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2023-11-01T00:15:20.170 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:15:20.170 INFO:teuthology.orchestra.run.smithi072.stdout:Upgraded: 2023-11-01T00:15:20.170 INFO:teuthology.orchestra.run.smithi072.stdout: librados2-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:15:20.170 INFO:teuthology.orchestra.run.smithi072.stdout: librbd1-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:15:20.171 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-11-01T00:15:20.171 INFO:teuthology.orchestra.run.smithi072.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-11-01T00:15:20.171 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-base-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:15:20.171 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-common-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:15:20.171 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-radosgw-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:15:20.172 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-selinux-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:15:20.172 INFO:teuthology.orchestra.run.smithi072.stdout: fmt-8.1.1-5.el9.x86_64 2023-11-01T00:15:20.172 INFO:teuthology.orchestra.run.smithi072.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-11-01T00:15:20.172 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-9.0.0-9.el9.x86_64 2023-11-01T00:15:20.172 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-11-01T00:15:20.173 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs2-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:15:20.173 INFO:teuthology.orchestra.run.smithi072.stdout: liboath-2.6.7-2.el9.x86_64 2023-11-01T00:15:20.173 INFO:teuthology.orchestra.run.smithi072.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-11-01T00:15:20.173 INFO:teuthology.orchestra.run.smithi072.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-11-01T00:15:20.173 INFO:teuthology.orchestra.run.smithi072.stdout: libradosstriper1-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:15:20.174 INFO:teuthology.orchestra.run.smithi072.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-11-01T00:15:20.174 INFO:teuthology.orchestra.run.smithi072.stdout: librgw2-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:15:20.174 INFO:teuthology.orchestra.run.smithi072.stdout: libunwind-1.6.2-1.el9.x86_64 2023-11-01T00:15:20.174 INFO:teuthology.orchestra.run.smithi072.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-11-01T00:15:20.175 INFO:teuthology.orchestra.run.smithi072.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-11-01T00:15:20.175 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-argparse-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:15:20.175 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-common-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:15:20.175 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cephfs-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:15:20.175 INFO:teuthology.orchestra.run.smithi072.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-11-01T00:15:20.176 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rados-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:15:20.176 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rbd-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:15:20.176 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rgw-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:15:20.176 INFO:teuthology.orchestra.run.smithi072.stdout: re2-1:20211101-3.el9.x86_64 2023-11-01T00:15:20.176 INFO:teuthology.orchestra.run.smithi072.stdout: thrift-0.15.0-2.el9.x86_64 2023-11-01T00:15:20.177 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:15:20.177 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:15:20.536 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-test 2023-11-01T00:15:21.077 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:00:32 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:15:21.224 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:15:21.224 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:15:21.225 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2023-11-01T00:15:21.225 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:15:21.225 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-11-01T00:15:21.225 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-test x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 117 M 2023-11-01T00:15:21.226 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2023-11-01T00:15:21.226 INFO:teuthology.orchestra.run.smithi072.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2023-11-01T00:15:21.226 INFO:teuthology.orchestra.run.smithi072.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-11-01T00:15:21.226 INFO:teuthology.orchestra.run.smithi072.stdout: libasan x86_64 11.4.1-2.3.el9 CentOS-AppStream 418 k 2023-11-01T00:15:21.227 INFO:teuthology.orchestra.run.smithi072.stdout: libcephsqlite x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 161 k 2023-11-01T00:15:21.227 INFO:teuthology.orchestra.run.smithi072.stdout: libubsan x86_64 11.4.1-2.3.el9 CentOS-AppStream 186 k 2023-11-01T00:15:21.227 INFO:teuthology.orchestra.run.smithi072.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-11-01T00:15:21.227 INFO:teuthology.orchestra.run.smithi072.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-11-01T00:15:21.227 INFO:teuthology.orchestra.run.smithi072.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-11-01T00:15:21.228 INFO:teuthology.orchestra.run.smithi072.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2023-11-01T00:15:21.228 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:15:21.228 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-11-01T00:15:21.228 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:15:21.228 INFO:teuthology.orchestra.run.smithi072.stdout:Install 10 Packages 2023-11-01T00:15:21.228 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:15:21.229 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 120 M 2023-11-01T00:15:21.229 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 1.1 G 2023-11-01T00:15:21.230 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-11-01T00:15:21.514 INFO:teuthology.orchestra.run.smithi072.stdout:(1/10): libasan-11.4.1-2.3.el9.x86_64.rpm 2.6 MB/s | 418 kB 00:00 2023-11-01T00:15:21.548 INFO:teuthology.orchestra.run.smithi072.stdout:(2/10): libubsan-11.4.1-2.3.el9.x86_64.rpm 5.4 MB/s | 186 kB 00:00 2023-11-01T00:15:21.582 INFO:teuthology.orchestra.run.smithi072.stdout:(3/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 6.3 MB/s | 217 kB 00:00 2023-11-01T00:15:21.625 INFO:teuthology.orchestra.run.smithi072.stdout:(4/10): socat-1.7.4.1-5.el9.x86_64.rpm 6.8 MB/s | 300 kB 00:00 2023-11-01T00:15:21.659 INFO:teuthology.orchestra.run.smithi072.stdout:(5/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-11-01T00:15:21.685 INFO:teuthology.orchestra.run.smithi072.stdout:(6/10): libcephsqlite-18.0.0-6795.gd1dcabba.el9 491 kB/s | 161 kB 00:00 2023-11-01T00:15:21.844 INFO:teuthology.orchestra.run.smithi072.stdout:(7/10): jq-1.6-15.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2023-11-01T00:15:22.078 INFO:teuthology.orchestra.run.smithi072.stdout:(8/10): cryptopp-8.6.0-1.el9.x86_64.rpm 3.2 MB/s | 1.3 MB 00:00 2023-11-01T00:15:22.103 INFO:teuthology.orchestra.run.smithi072.stdout:(9/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 471 kB/s | 122 kB 00:00 2023-11-01T00:15:27.345 INFO:teuthology.orchestra.run.smithi072.stdout:(10/10): ceph-test-18.0.0-6795.gd1dcabba.el9.x8 20 MB/s | 117 MB 00:05 2023-11-01T00:15:27.350 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-11-01T00:15:27.350 INFO:teuthology.orchestra.run.smithi072.stdout:Total 20 MB/s | 120 MB 00:06 2023-11-01T00:15:27.363 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-11-01T00:15:27.384 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-11-01T00:15:27.384 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-11-01T00:15:28.205 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-11-01T00:15:28.205 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-11-01T00:15:29.299 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-11-01T00:15:29.388 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2023-11-01T00:15:29.430 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2023-11-01T00:15:29.475 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/10 2023-11-01T00:15:29.519 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/10 2023-11-01T00:15:29.559 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-11-01T00:15:29.592 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : jq-1.6-15.el9.x86_64 6/10 2023-11-01T00:15:29.639 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libubsan-11.4.1-2.3.el9.x86_64 7/10 2023-11-01T00:15:29.688 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libasan-11.4.1-2.3.el9.x86_64 8/10 2023-11-01T00:15:29.716 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephsqlite-2:18.0.0-6795.gd1dcabba.el9.x86_64 9/10 2023-11-01T00:15:45.950 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6795.gd1dcabba.el9.x86_64 9/10 2023-11-01T00:15:46.029 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-test-2:18.0.0-6795.gd1dcabba.el9.x86_64 10/10 2023-11-01T00:16:01.757 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-test-2:18.0.0-6795.gd1dcabba.el9.x86_64 10/10 2023-11-01T00:16:01.758 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-test-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/10 2023-11-01T00:16:01.758 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephsqlite-2:18.0.0-6795.gd1dcabba.el9.x86_64 2/10 2023-11-01T00:16:01.758 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libasan-11.4.1-2.3.el9.x86_64 3/10 2023-11-01T00:16:01.758 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libubsan-11.4.1-2.3.el9.x86_64 4/10 2023-11-01T00:16:01.759 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 5/10 2023-11-01T00:16:01.759 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 6/10 2023-11-01T00:16:01.759 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 7/10 2023-11-01T00:16:01.759 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : jq-1.6-15.el9.x86_64 8/10 2023-11-01T00:16:01.760 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2023-11-01T00:16:02.235 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2023-11-01T00:16:02.235 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:16:02.236 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-11-01T00:16:02.236 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-test-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:16:02.236 INFO:teuthology.orchestra.run.smithi072.stdout: cryptopp-8.6.0-1.el9.x86_64 2023-11-01T00:16:02.236 INFO:teuthology.orchestra.run.smithi072.stdout: jq-1.6-15.el9.x86_64 2023-11-01T00:16:02.236 INFO:teuthology.orchestra.run.smithi072.stdout: libasan-11.4.1-2.3.el9.x86_64 2023-11-01T00:16:02.237 INFO:teuthology.orchestra.run.smithi072.stdout: libcephsqlite-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:16:02.237 INFO:teuthology.orchestra.run.smithi072.stdout: libubsan-11.4.1-2.3.el9.x86_64 2023-11-01T00:16:02.237 INFO:teuthology.orchestra.run.smithi072.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-11-01T00:16:02.237 INFO:teuthology.orchestra.run.smithi072.stdout: socat-1.7.4.1-5.el9.x86_64 2023-11-01T00:16:02.237 INFO:teuthology.orchestra.run.smithi072.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-11-01T00:16:02.238 INFO:teuthology.orchestra.run.smithi072.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2023-11-01T00:16:02.238 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:16:02.238 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:16:02.609 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph 2023-11-01T00:16:03.157 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:14 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:16:03.301 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:16:03.303 INFO:teuthology.orchestra.run.smithi072.stdout:========================================================================================= 2023-11-01T00:16:03.303 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2023-11-01T00:16:03.304 INFO:teuthology.orchestra.run.smithi072.stdout:========================================================================================= 2023-11-01T00:16:03.304 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-11-01T00:16:03.304 INFO:teuthology.orchestra.run.smithi072.stdout: ceph x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 6.5 k 2023-11-01T00:16:03.304 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2023-11-01T00:16:03.305 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mds x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 2.0 M 2023-11-01T00:16:03.305 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 1.5 M 2023-11-01T00:16:03.305 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6795.gd1dcabba.el9 ceph-noarch 243 k 2023-11-01T00:16:03.305 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mon x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 8.8 M 2023-11-01T00:16:03.305 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-osd x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 117 M 2023-11-01T00:16:03.306 INFO:teuthology.orchestra.run.smithi072.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-11-01T00:16:03.306 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-11-01T00:16:03.306 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-11-01T00:16:03.306 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-11-01T00:16:03.306 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-11-01T00:16:03.307 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-11-01T00:16:03.307 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-11-01T00:16:03.307 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-11-01T00:16:03.307 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-11-01T00:16:03.307 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-11-01T00:16:03.308 INFO:teuthology.orchestra.run.smithi072.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-11-01T00:16:03.308 INFO:teuthology.orchestra.run.smithi072.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-11-01T00:16:03.308 INFO:teuthology.orchestra.run.smithi072.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-11-01T00:16:03.308 INFO:teuthology.orchestra.run.smithi072.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-11-01T00:16:03.308 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-11-01T00:16:03.309 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-11-01T00:16:03.309 INFO:teuthology.orchestra.run.smithi072.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-11-01T00:16:03.309 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-11-01T00:16:03.309 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-11-01T00:16:03.309 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-11-01T00:16:03.310 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-11-01T00:16:03.310 INFO:teuthology.orchestra.run.smithi072.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-11-01T00:16:03.310 INFO:teuthology.orchestra.run.smithi072.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-11-01T00:16:03.310 INFO:teuthology.orchestra.run.smithi072.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-11-01T00:16:03.310 INFO:teuthology.orchestra.run.smithi072.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-11-01T00:16:03.311 INFO:teuthology.orchestra.run.smithi072.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-11-01T00:16:03.311 INFO:teuthology.orchestra.run.smithi072.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-11-01T00:16:03.311 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:16:03.311 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-11-01T00:16:03.311 INFO:teuthology.orchestra.run.smithi072.stdout:========================================================================================= 2023-11-01T00:16:03.312 INFO:teuthology.orchestra.run.smithi072.stdout:Install 33 Packages 2023-11-01T00:16:03.312 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:16:03.312 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 133 M 2023-11-01T00:16:03.313 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 1.4 G 2023-11-01T00:16:03.313 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-11-01T00:16:03.930 INFO:teuthology.orchestra.run.smithi072.stdout:(1/33): ceph-18.0.0-6795.gd1dcabba.el9.x86_64.r 25 kB/s | 6.5 kB 00:00 2023-11-01T00:16:04.714 INFO:teuthology.orchestra.run.smithi072.stdout:(2/33): ceph-mgr-18.0.0-6795.gd1dcabba.el9.x86_ 1.4 MB/s | 1.5 MB 00:01 2023-11-01T00:16:04.822 INFO:teuthology.orchestra.run.smithi072.stdout:(3/33): ceph-mds-18.0.0-6795.gd1dcabba.el9.x86_ 1.7 MB/s | 2.0 MB 00:01 2023-11-01T00:16:04.956 INFO:teuthology.orchestra.run.smithi072.stdout:(4/33): ceph-mgr-modules-core-18.0.0-6795.gd1dc 1.8 MB/s | 243 kB 00:00 2023-11-01T00:16:05.057 INFO:teuthology.orchestra.run.smithi072.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 144 kB 00:00 2023-11-01T00:16:05.091 INFO:teuthology.orchestra.run.smithi072.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 944 kB/s | 32 kB 00:00 2023-11-01T00:16:05.124 INFO:teuthology.orchestra.run.smithi072.stdout:(7/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-11-01T00:16:05.175 INFO:teuthology.orchestra.run.smithi072.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-11-01T00:16:05.209 INFO:teuthology.orchestra.run.smithi072.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2023-11-01T00:16:05.243 INFO:teuthology.orchestra.run.smithi072.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2023-11-01T00:16:05.276 INFO:teuthology.orchestra.run.smithi072.stdout:(11/33): python3-requests-2.25.1-7.el9.noarch.r 3.3 MB/s | 113 kB 00:00 2023-11-01T00:16:05.319 INFO:teuthology.orchestra.run.smithi072.stdout:(12/33): python3-urllib3-1.26.5-4.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2023-11-01T00:16:05.628 INFO:teuthology.orchestra.run.smithi072.stdout:(13/33): python3-cryptography-36.0.1-4.el9.x86_ 4.0 MB/s | 1.2 MB 00:00 2023-11-01T00:16:05.720 INFO:teuthology.orchestra.run.smithi072.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 556 kB/s | 51 kB 00:00 2023-11-01T00:16:05.854 INFO:teuthology.orchestra.run.smithi072.stdout:(15/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 325 kB/s | 43 kB 00:00 2023-11-01T00:16:05.964 INFO:teuthology.orchestra.run.smithi072.stdout:(16/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2023-11-01T00:16:06.056 INFO:teuthology.orchestra.run.smithi072.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2023-11-01T00:16:06.107 INFO:teuthology.orchestra.run.smithi072.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2023-11-01T00:16:06.149 INFO:teuthology.orchestra.run.smithi072.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 421 kB/s | 18 kB 00:00 2023-11-01T00:16:06.191 INFO:teuthology.orchestra.run.smithi072.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 551 kB/s | 23 kB 00:00 2023-11-01T00:16:06.233 INFO:teuthology.orchestra.run.smithi072.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 463 kB/s | 19 kB 00:00 2023-11-01T00:16:06.276 INFO:teuthology.orchestra.run.smithi072.stdout:(22/33): python3-jaraco-text-3.2.0-6.el9.noarch 467 kB/s | 20 kB 00:00 2023-11-01T00:16:06.460 INFO:teuthology.orchestra.run.smithi072.stdout:(23/33): ceph-mon-18.0.0-6795.gd1dcabba.el9.x86 3.5 MB/s | 8.8 MB 00:02 2023-11-01T00:16:06.485 INFO:teuthology.orchestra.run.smithi072.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 221 kB/s | 46 kB 00:00 2023-11-01T00:16:06.577 INFO:teuthology.orchestra.run.smithi072.stdout:(25/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2023-11-01T00:16:06.611 INFO:teuthology.orchestra.run.smithi072.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 521 kB/s | 79 kB 00:00 2023-11-01T00:16:06.637 INFO:teuthology.orchestra.run.smithi072.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 278 kB/s | 16 kB 00:00 2023-11-01T00:16:06.687 INFO:teuthology.orchestra.run.smithi072.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2023-11-01T00:16:06.713 INFO:teuthology.orchestra.run.smithi072.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 471 kB/s | 36 kB 00:00 2023-11-01T00:16:06.780 INFO:teuthology.orchestra.run.smithi072.stdout:(30/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 6.3 MB/s | 427 kB 00:00 2023-11-01T00:16:06.807 INFO:teuthology.orchestra.run.smithi072.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2023-11-01T00:16:06.841 INFO:teuthology.orchestra.run.smithi072.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 330 kB/s | 20 kB 00:00 2023-11-01T00:16:13.258 INFO:teuthology.orchestra.run.smithi072.stdout:(33/33): ceph-osd-18.0.0-6795.gd1dcabba.el9.x86 14 MB/s | 117 MB 00:08 2023-11-01T00:16:13.262 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-11-01T00:16:13.262 INFO:teuthology.orchestra.run.smithi072.stdout:Total 13 MB/s | 133 MB 00:09 2023-11-01T00:16:13.295 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-11-01T00:16:13.333 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-11-01T00:16:13.334 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-11-01T00:16:14.317 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-11-01T00:16:14.318 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-11-01T00:16:16.156 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-11-01T00:16:16.228 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2023-11-01T00:16:16.268 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2023-11-01T00:16:16.302 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2023-11-01T00:16:16.393 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2023-11-01T00:16:16.472 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2023-11-01T00:16:16.516 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2023-11-01T00:16:16.554 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2023-11-01T00:16:16.631 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 8/33 2023-11-01T00:16:16.685 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 9/33 2023-11-01T00:16:16.725 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 10/33 2023-11-01T00:16:16.774 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ply-3.11-14.el9.noarch 11/33 2023-11-01T00:16:16.838 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 12/33 2023-11-01T00:16:17.028 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 13/33 2023-11-01T00:16:17.085 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 14/33 2023-11-01T00:16:17.131 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 15/33 2023-11-01T00:16:17.176 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 16/33 2023-11-01T00:16:17.215 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 17/33 2023-11-01T00:16:17.291 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 18/33 2023-11-01T00:16:17.327 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 19/33 2023-11-01T00:16:17.352 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 20/33 2023-11-01T00:16:17.396 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 21/33 2023-11-01T00:16:17.431 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2023-11-01T00:16:17.519 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2023-11-01T00:16:17.561 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2023-11-01T00:16:17.643 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2023-11-01T00:16:17.781 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2023-11-01T00:16:17.908 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-11-01T00:16:18.077 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6795.gd1dcabba.el9. 28/33 2023-11-01T00:16:18.108 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-2:18.0.0-6795.gd1dcabba.el9.x86_64 29/33 2023-11-01T00:16:18.133 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6795.gd1dcabba.el9.x86_64 29/33 2023-11-01T00:16:18.133 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-01T00:16:18.133 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-11-01T00:16:18.134 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-01T00:16:18.134 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-11-01T00:16:18.134 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:16:36.303 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-osd-2:18.0.0-6795.gd1dcabba.el9.x86_64 30/33 2023-11-01T00:16:36.328 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-osd-2:18.0.0-6795.gd1dcabba.el9.x86_64 30/33 2023-11-01T00:16:36.329 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-01T00:16:36.329 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-11-01T00:16:36.329 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-01T00:16:36.329 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-11-01T00:16:36.330 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:16:37.204 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mon-2:18.0.0-6795.gd1dcabba.el9.x86_64 31/33 2023-11-01T00:16:37.227 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mon-2:18.0.0-6795.gd1dcabba.el9.x86_64 31/33 2023-11-01T00:16:37.227 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-01T00:16:37.227 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-11-01T00:16:37.227 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-01T00:16:37.227 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-11-01T00:16:37.228 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:16:37.430 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mds-2:18.0.0-6795.gd1dcabba.el9.x86_64 32/33 2023-11-01T00:16:37.453 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mds-2:18.0.0-6795.gd1dcabba.el9.x86_64 32/33 2023-11-01T00:16:37.453 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-01T00:16:37.453 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-11-01T00:16:37.453 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-01T00:16:37.454 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-11-01T00:16:37.454 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:16:37.522 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-2:18.0.0-6795.gd1dcabba.el9.x86_64 33/33 2023-11-01T00:16:56.849 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-2:18.0.0-6795.gd1dcabba.el9.x86_64 33/33 2023-11-01T00:16:56.849 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/33 2023-11-01T00:16:56.850 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mds-2:18.0.0-6795.gd1dcabba.el9.x86_64 2/33 2023-11-01T00:16:56.850 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-2:18.0.0-6795.gd1dcabba.el9.x86_64 3/33 2023-11-01T00:16:56.850 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mon-2:18.0.0-6795.gd1dcabba.el9.x86_64 4/33 2023-11-01T00:16:56.850 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-osd-2:18.0.0-6795.gd1dcabba.el9.x86_64 5/33 2023-11-01T00:16:56.850 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6795.gd1dcabba.el9. 6/33 2023-11-01T00:16:56.851 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2023-11-01T00:16:56.851 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2023-11-01T00:16:56.851 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 9/33 2023-11-01T00:16:56.851 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 10/33 2023-11-01T00:16:56.851 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ply-3.11-14.el9.noarch 11/33 2023-11-01T00:16:56.852 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 12/33 2023-11-01T00:16:56.852 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 13/33 2023-11-01T00:16:56.852 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 14/33 2023-11-01T00:16:56.852 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 15/33 2023-11-01T00:16:56.852 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/33 2023-11-01T00:16:56.853 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2023-11-01T00:16:56.853 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2023-11-01T00:16:56.853 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2023-11-01T00:16:56.853 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2023-11-01T00:16:56.853 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2023-11-01T00:16:56.853 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2023-11-01T00:16:56.854 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2023-11-01T00:16:56.854 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2023-11-01T00:16:56.854 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2023-11-01T00:16:56.854 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2023-11-01T00:16:56.854 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2023-11-01T00:16:56.855 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2023-11-01T00:16:56.855 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2023-11-01T00:16:56.856 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2023-11-01T00:16:56.856 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2023-11-01T00:16:56.856 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2023-11-01T00:16:57.738 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2023-11-01T00:16:57.738 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:16:57.738 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-11-01T00:16:57.739 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:16:57.739 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mds-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:16:57.739 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:16:57.739 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-modules-core-2:18.0.0-6795.gd1dcabba.el9.noarch 2023-11-01T00:16:57.739 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mon-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:16:57.740 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-osd-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:16:57.740 INFO:teuthology.orchestra.run.smithi072.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-11-01T00:16:57.740 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-11-01T00:16:57.740 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-11-01T00:16:57.740 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-11-01T00:16:57.740 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-11-01T00:16:57.741 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-11-01T00:16:57.741 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-11-01T00:16:57.741 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-11-01T00:16:57.741 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-11-01T00:16:57.741 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-11-01T00:16:57.741 INFO:teuthology.orchestra.run.smithi072.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-11-01T00:16:57.742 INFO:teuthology.orchestra.run.smithi072.stdout: python3-mako-1.1.4-6.el9.noarch 2023-11-01T00:16:57.742 INFO:teuthology.orchestra.run.smithi072.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-11-01T00:16:57.742 INFO:teuthology.orchestra.run.smithi072.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-11-01T00:16:57.742 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-11-01T00:16:57.743 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ply-3.11-14.el9.noarch 2023-11-01T00:16:57.743 INFO:teuthology.orchestra.run.smithi072.stdout: python3-portend-3.1.0-2.el9.noarch 2023-11-01T00:16:57.743 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-11-01T00:16:57.743 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pycparser-2.20-6.el9.noarch 2023-11-01T00:16:57.743 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pytz-2021.1-5.el9.noarch 2023-11-01T00:16:57.743 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-2.25.1-7.el9.noarch 2023-11-01T00:16:57.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-11-01T00:16:57.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-toml-0.10.2-6.el9.noarch 2023-11-01T00:16:57.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-11-01T00:16:57.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-webob-1.8.7-6.el9.noarch 2023-11-01T00:16:57.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-11-01T00:16:57.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-11-01T00:16:57.745 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:16:57.745 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:16:58.119 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-base 2023-11-01T00:16:58.682 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:09 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:16:58.777 INFO:teuthology.orchestra.run.smithi072.stdout:Package ceph-base-2:18.0.0-6795.gd1dcabba.el9.x86_64 is already installed. 2023-11-01T00:16:58.828 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:16:58.828 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2023-11-01T00:16:58.829 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:16:58.918 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install cephadm 2023-11-01T00:16:59.449 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:10 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:16:59.588 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:16:59.589 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:16:59.589 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2023-11-01T00:16:59.589 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:16:59.589 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-11-01T00:16:59.590 INFO:teuthology.orchestra.run.smithi072.stdout: cephadm noarch 2:18.0.0-6795.gd1dcabba.el9 ceph-noarch 270 k 2023-11-01T00:16:59.590 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:16:59.590 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-11-01T00:16:59.590 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:16:59.590 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2023-11-01T00:16:59.591 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:16:59.591 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 270 k 2023-11-01T00:16:59.591 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 265 k 2023-11-01T00:16:59.591 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-11-01T00:16:59.818 INFO:teuthology.orchestra.run.smithi072.stdout:cephadm-18.0.0-6795.gd1dcabba.el9.noarch.rpm 1.2 MB/s | 270 kB 00:00 2023-11-01T00:16:59.819 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-11-01T00:16:59.819 INFO:teuthology.orchestra.run.smithi072.stdout:Total 1.1 MB/s | 270 kB 00:00 2023-11-01T00:16:59.819 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-11-01T00:16:59.824 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-11-01T00:16:59.824 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-11-01T00:16:59.827 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-11-01T00:16:59.827 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-11-01T00:16:59.898 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-11-01T00:17:00.344 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: cephadm-2:18.0.0-6795.gd1dcabba.el9.noarch 1/1 2023-11-01T00:17:00.435 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephadm-2:18.0.0-6795.gd1dcabba.el9.noarch 1/1 2023-11-01T00:17:00.870 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: cephadm-2:18.0.0-6795.gd1dcabba.el9.noarch 1/1 2023-11-01T00:17:01.157 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephadm-2:18.0.0-6795.gd1dcabba.el9.noarch 1/1 2023-11-01T00:17:01.157 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:01.158 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-11-01T00:17:01.158 INFO:teuthology.orchestra.run.smithi072.stdout: cephadm-2:18.0.0-6795.gd1dcabba.el9.noarch 2023-11-01T00:17:01.158 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:01.159 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:17:01.390 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-immutable-object-cache 2023-11-01T00:17:01.934 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:12 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:17:02.077 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:17:02.078 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:02.078 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repo Size 2023-11-01T00:17:02.078 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:02.079 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-11-01T00:17:02.079 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 144 k 2023-11-01T00:17:02.079 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:02.079 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-11-01T00:17:02.079 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:02.080 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2023-11-01T00:17:02.080 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:02.080 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 144 k 2023-11-01T00:17:02.080 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 437 k 2023-11-01T00:17:02.080 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-11-01T00:17:02.302 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-immutable-object-cache-18.0.0-6795.gd1dcab 647 kB/s | 144 kB 00:00 2023-11-01T00:17:02.303 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-11-01T00:17:02.303 INFO:teuthology.orchestra.run.smithi072.stdout:Total 643 kB/s | 144 kB 00:00 2023-11-01T00:17:02.303 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-11-01T00:17:02.310 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-11-01T00:17:02.310 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-11-01T00:17:02.361 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-11-01T00:17:02.362 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-11-01T00:17:02.514 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-11-01T00:17:02.620 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6795.gd1dcabba. 1/1 2023-11-01T00:17:02.644 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6795.gd1dcabba. 1/1 2023-11-01T00:17:02.645 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-01T00:17:02.645 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-11-01T00:17:02.645 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:03.657 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6795.gd1dcabba. 1/1 2023-11-01T00:17:03.657 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:03.658 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-11-01T00:17:03.658 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-immutable-object-cache-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:17:03.658 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:03.658 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:17:03.929 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr 2023-11-01T00:17:04.480 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:15 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:17:04.570 INFO:teuthology.orchestra.run.smithi072.stdout:Package ceph-mgr-2:18.0.0-6795.gd1dcabba.el9.x86_64 is already installed. 2023-11-01T00:17:04.618 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:17:04.619 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2023-11-01T00:17:04.619 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:17:04.711 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr-dashboard 2023-11-01T00:17:05.249 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:16 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:17:05.389 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:17:05.390 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:05.390 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2023-11-01T00:17:05.390 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:05.390 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-11-01T00:17:05.391 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6795.gd1dcabba.el9 ceph-noarch 1.8 M 2023-11-01T00:17:05.391 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2023-11-01T00:17:05.391 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6795.gd1dcabba.el9 ceph-noarch 23 k 2023-11-01T00:17:05.391 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6795.gd1dcabba.el9 ceph-noarch 15 k 2023-11-01T00:17:05.391 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-11-01T00:17:05.392 INFO:teuthology.orchestra.run.smithi072.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-11-01T00:17:05.392 INFO:teuthology.orchestra.run.smithi072.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-11-01T00:17:05.392 INFO:teuthology.orchestra.run.smithi072.stdout:Installing weak dependencies: 2023-11-01T00:17:05.392 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-11-01T00:17:05.392 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:05.393 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-11-01T00:17:05.393 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:05.393 INFO:teuthology.orchestra.run.smithi072.stdout:Install 7 Packages 2023-11-01T00:17:05.393 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:05.394 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 2.1 M 2023-11-01T00:17:05.394 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 8.6 M 2023-11-01T00:17:05.394 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-11-01T00:17:05.745 INFO:teuthology.orchestra.run.smithi072.stdout:(1/7): ceph-grafana-dashboards-18.0.0-6795.gd1d 118 kB/s | 23 kB 00:00 2023-11-01T00:17:05.789 INFO:teuthology.orchestra.run.smithi072.stdout:(2/7): ceph-prometheus-alerts-18.0.0-6795.gd1dc 63 kB/s | 15 kB 00:00 2023-11-01T00:17:05.948 INFO:teuthology.orchestra.run.smithi072.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 45 kB/s | 9.0 kB 00:00 2023-11-01T00:17:05.998 INFO:teuthology.orchestra.run.smithi072.stdout:(4/7): ceph-mgr-dashboard-18.0.0-6795.gd1dcabba 3.9 MB/s | 1.8 MB 00:00 2023-11-01T00:17:06.057 INFO:teuthology.orchestra.run.smithi072.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 281 kB/s | 31 kB 00:00 2023-11-01T00:17:06.107 INFO:teuthology.orchestra.run.smithi072.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 128 kB/s | 41 kB 00:00 2023-11-01T00:17:06.383 INFO:teuthology.orchestra.run.smithi072.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 491 kB/s | 188 kB 00:00 2023-11-01T00:17:06.387 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-11-01T00:17:06.387 INFO:teuthology.orchestra.run.smithi072.stdout:Total 2.1 MB/s | 2.1 MB 00:00 2023-11-01T00:17:06.388 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-11-01T00:17:06.395 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-11-01T00:17:06.395 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-11-01T00:17:06.460 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-11-01T00:17:06.460 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-11-01T00:17:06.754 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-11-01T00:17:06.827 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-11-01T00:17:06.860 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-11-01T00:17:06.907 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-11-01T00:17:06.940 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-11-01T00:17:06.977 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6795.gd1dcabba.el9.n 5/7 2023-11-01T00:17:07.213 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6795.gd1dcabba.el9. 6/7 2023-11-01T00:17:07.250 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6795.gd1dcabba.el9.noarc 7/7 2023-11-01T00:17:07.892 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6795.gd1dcabba.el9.noarc 7/7 2023-11-01T00:17:07.892 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6795.gd1dcabba.el9. 1/7 2023-11-01T00:17:07.892 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6795.gd1dcabba.el9.noarc 2/7 2023-11-01T00:17:07.892 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6795.gd1dcabba.el9.n 3/7 2023-11-01T00:17:07.892 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-11-01T00:17:07.893 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-11-01T00:17:07.893 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-11-01T00:17:08.216 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-11-01T00:17:08.217 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:08.217 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-11-01T00:17:08.217 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-grafana-dashboards-2:18.0.0-6795.gd1dcabba.el9.noarch 2023-11-01T00:17:08.217 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-dashboard-2:18.0.0-6795.gd1dcabba.el9.noarch 2023-11-01T00:17:08.218 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-prometheus-alerts-2:18.0.0-6795.gd1dcabba.el9.noarch 2023-11-01T00:17:08.218 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-11-01T00:17:08.218 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-11-01T00:17:08.218 INFO:teuthology.orchestra.run.smithi072.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-11-01T00:17:08.218 INFO:teuthology.orchestra.run.smithi072.stdout: python3-routes-2.5.1-5.el9.noarch 2023-11-01T00:17:08.219 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:08.219 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:17:08.456 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-01T00:17:09.005 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:20 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:17:09.146 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:17:09.147 INFO:teuthology.orchestra.run.smithi072.stdout:============================================================================================ 2023-11-01T00:17:09.147 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2023-11-01T00:17:09.148 INFO:teuthology.orchestra.run.smithi072.stdout:============================================================================================ 2023-11-01T00:17:09.148 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-11-01T00:17:09.148 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6795.gd1dcabba.el9 ceph-noarch 7.4 M 2023-11-01T00:17:09.148 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2023-11-01T00:17:09.148 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-11-01T00:17:09.149 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-11-01T00:17:09.149 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-11-01T00:17:09.149 INFO:teuthology.orchestra.run.smithi072.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-11-01T00:17:09.149 INFO:teuthology.orchestra.run.smithi072.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-11-01T00:17:09.149 INFO:teuthology.orchestra.run.smithi072.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-11-01T00:17:09.150 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-11-01T00:17:09.150 INFO:teuthology.orchestra.run.smithi072.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-11-01T00:17:09.150 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-11-01T00:17:09.150 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-11-01T00:17:09.150 INFO:teuthology.orchestra.run.smithi072.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-11-01T00:17:09.151 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:09.151 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-11-01T00:17:09.151 INFO:teuthology.orchestra.run.smithi072.stdout:============================================================================================ 2023-11-01T00:17:09.151 INFO:teuthology.orchestra.run.smithi072.stdout:Install 12 Packages 2023-11-01T00:17:09.151 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:09.152 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 37 M 2023-11-01T00:17:09.152 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 209 M 2023-11-01T00:17:09.152 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-11-01T00:17:09.281 INFO:teuthology.orchestra.run.smithi072.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 241 kB/s | 32 kB 00:00 2023-11-01T00:17:09.316 INFO:teuthology.orchestra.run.smithi072.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 490 kB/s | 17 kB 00:00 2023-11-01T00:17:09.359 INFO:teuthology.orchestra.run.smithi072.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 847 kB/s | 34 kB 00:00 2023-11-01T00:17:09.442 INFO:teuthology.orchestra.run.smithi072.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2023-11-01T00:17:09.518 INFO:teuthology.orchestra.run.smithi072.stdout:(5/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.7 MB/s | 205 kB 00:00 2023-11-01T00:17:09.627 INFO:teuthology.orchestra.run.smithi072.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2023-11-01T00:17:09.744 INFO:teuthology.orchestra.run.smithi072.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2023-11-01T00:17:09.778 INFO:teuthology.orchestra.run.smithi072.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.4 MB/s | 217 kB 00:00 2023-11-01T00:17:10.104 INFO:teuthology.orchestra.run.smithi072.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2023-11-01T00:17:10.146 INFO:teuthology.orchestra.run.smithi072.stdout:(10/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2023-11-01T00:17:10.180 INFO:teuthology.orchestra.run.smithi072.stdout:(11/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 2.6 MB/s | 195 kB 00:00 2023-11-01T00:17:10.555 INFO:teuthology.orchestra.run.smithi072.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-6 5.3 MB/s | 7.4 MB 00:01 2023-11-01T00:17:10.558 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-11-01T00:17:10.559 INFO:teuthology.orchestra.run.smithi072.stdout:Total 26 MB/s | 37 MB 00:01 2023-11-01T00:17:10.559 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-11-01T00:17:10.581 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-11-01T00:17:10.581 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-11-01T00:17:11.022 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-11-01T00:17:11.023 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-11-01T00:17:11.719 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-11-01T00:17:11.787 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-11-01T00:17:11.827 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-11-01T00:17:11.938 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-11-01T00:17:11.975 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-11-01T00:17:12.161 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-01T00:17:12.191 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-01T00:17:12.282 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-11-01T00:17:12.935 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-11-01T00:17:13.054 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-11-01T00:17:14.273 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-11-01T00:17:15.311 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-11-01T00:17:15.340 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6795.gd1dca 12/12 2023-11-01T00:17:19.341 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6795.gd1dca 12/12 2023-11-01T00:17:19.341 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6795.gd1dca 1/12 2023-11-01T00:17:19.341 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-11-01T00:17:19.342 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-11-01T00:17:19.342 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-11-01T00:17:19.342 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-11-01T00:17:19.342 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-11-01T00:17:19.343 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 7/12 2023-11-01T00:17:19.343 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 8/12 2023-11-01T00:17:19.343 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 9/12 2023-11-01T00:17:19.343 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 10/12 2023-11-01T00:17:19.343 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 11/12 2023-11-01T00:17:19.805 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 12/12 2023-11-01T00:17:19.805 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:19.805 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-11-01T00:17:19.806 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6795.gd1dcabba.el9.noarch 2023-11-01T00:17:19.806 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-11-01T00:17:19.806 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-11-01T00:17:19.806 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-11-01T00:17:19.807 INFO:teuthology.orchestra.run.smithi072.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-11-01T00:17:19.807 INFO:teuthology.orchestra.run.smithi072.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-11-01T00:17:19.807 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-0.3.21-2.el9.x86_64 2023-11-01T00:17:19.807 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-11-01T00:17:19.807 INFO:teuthology.orchestra.run.smithi072.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-11-01T00:17:19.807 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-11-01T00:17:19.808 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-11-01T00:17:19.808 INFO:teuthology.orchestra.run.smithi072.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-11-01T00:17:19.808 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:19.808 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:17:20.317 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr-rook 2023-11-01T00:17:20.866 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:31 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:17:21.008 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:17:21.009 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================== 2023-11-01T00:17:21.009 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2023-11-01T00:17:21.009 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================== 2023-11-01T00:17:21.009 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-11-01T00:17:21.010 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-rook noarch 2:18.0.0-6795.gd1dcabba.el9 ceph-noarch 48 k 2023-11-01T00:17:21.010 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2023-11-01T00:17:21.010 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-11-01T00:17:21.010 INFO:teuthology.orchestra.run.smithi072.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-11-01T00:17:21.010 INFO:teuthology.orchestra.run.smithi072.stdout: python3-google-auth noarch 1:2.23.3-1.el9 epel 197 k 2023-11-01T00:17:21.011 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-11-01T00:17:21.011 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-11-01T00:17:21.011 INFO:teuthology.orchestra.run.smithi072.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-11-01T00:17:21.011 INFO:teuthology.orchestra.run.smithi072.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-11-01T00:17:21.011 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-11-01T00:17:21.012 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-11-01T00:17:21.012 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-11-01T00:17:21.012 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-11-01T00:17:21.012 INFO:teuthology.orchestra.run.smithi072.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-11-01T00:17:21.012 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:21.012 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-11-01T00:17:21.013 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================== 2023-11-01T00:17:21.013 INFO:teuthology.orchestra.run.smithi072.stdout:Install 13 Packages 2023-11-01T00:17:21.013 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:21.014 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 2.0 M 2023-11-01T00:17:21.014 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 26 M 2023-11-01T00:17:21.014 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-11-01T00:17:21.295 INFO:teuthology.orchestra.run.smithi072.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 202 kB/s | 24 kB 00:00 2023-11-01T00:17:21.305 INFO:teuthology.orchestra.run.smithi072.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 138 kB/s | 18 kB 00:00 2023-11-01T00:17:21.347 INFO:teuthology.orchestra.run.smithi072.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 3.3 MB/s | 174 kB 00:00 2023-11-01T00:17:21.373 INFO:teuthology.orchestra.run.smithi072.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-11-01T00:17:21.398 INFO:teuthology.orchestra.run.smithi072.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-11-01T00:17:21.424 INFO:teuthology.orchestra.run.smithi072.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 845 kB/s | 43 kB 00:00 2023-11-01T00:17:21.450 INFO:teuthology.orchestra.run.smithi072.stdout:(7/13): ceph-mgr-rook-18.0.0-6795.gd1dcabba.el9 173 kB/s | 48 kB 00:00 2023-11-01T00:17:21.876 INFO:teuthology.orchestra.run.smithi072.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 68 kB/s | 32 kB 00:00 2023-11-01T00:17:21.901 INFO:teuthology.orchestra.run.smithi072.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 31 kB/s | 15 kB 00:00 2023-11-01T00:17:22.085 INFO:teuthology.orchestra.run.smithi072.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 322 kB/s | 59 kB 00:00 2023-11-01T00:17:22.169 INFO:teuthology.orchestra.run.smithi072.stdout:(11/13): python3-google-auth-2.23.3-1.el9.noarc 274 kB/s | 197 kB 00:00 2023-11-01T00:17:22.269 INFO:teuthology.orchestra.run.smithi072.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 487 kB/s | 90 kB 00:00 2023-11-01T00:17:23.278 INFO:teuthology.orchestra.run.smithi072.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 746 kB/s | 1.0 MB 00:01 2023-11-01T00:17:23.283 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-11-01T00:17:23.283 INFO:teuthology.orchestra.run.smithi072.stdout:Total 919 kB/s | 2.0 MB 00:02 2023-11-01T00:17:23.284 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-11-01T00:17:23.298 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-11-01T00:17:23.299 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-11-01T00:17:23.400 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-11-01T00:17:23.401 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-11-01T00:17:23.924 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-11-01T00:17:24.102 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-11-01T00:17:24.157 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-11-01T00:17:24.221 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-11-01T00:17:24.260 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-11-01T00:17:24.301 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-11-01T00:17:24.394 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-11-01T00:17:24.507 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-google-auth-1:2.23.3-1.el9.noarch 7/13 2023-11-01T00:17:24.574 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-11-01T00:17:25.293 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-11-01T00:17:25.351 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-11-01T00:17:25.385 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-11-01T00:17:25.440 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-11-01T00:17:25.473 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-rook-2:18.0.0-6795.gd1dcabba.el9.noarch 13/13 2023-11-01T00:17:26.530 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6795.gd1dcabba.el9.noarch 13/13 2023-11-01T00:17:26.530 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6795.gd1dcabba.el9.noarch 1/13 2023-11-01T00:17:26.530 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-11-01T00:17:26.530 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-11-01T00:17:26.531 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-11-01T00:17:26.531 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-11-01T00:17:26.531 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-11-01T00:17:26.531 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-11-01T00:17:26.531 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-11-01T00:17:26.532 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-11-01T00:17:26.532 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-google-auth-1:2.23.3-1.el9.noarch 10/13 2023-11-01T00:17:26.532 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-11-01T00:17:26.532 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-11-01T00:17:26.902 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-11-01T00:17:26.903 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:26.903 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-11-01T00:17:26.903 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-rook-2:18.0.0-6795.gd1dcabba.el9.noarch 2023-11-01T00:17:26.903 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-11-01T00:17:26.903 INFO:teuthology.orchestra.run.smithi072.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-11-01T00:17:26.904 INFO:teuthology.orchestra.run.smithi072.stdout: python3-google-auth-1:2.23.3-1.el9.noarch 2023-11-01T00:17:26.904 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-11-01T00:17:26.904 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-11-01T00:17:26.904 INFO:teuthology.orchestra.run.smithi072.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-11-01T00:17:26.904 INFO:teuthology.orchestra.run.smithi072.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-11-01T00:17:26.905 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-11-01T00:17:26.905 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-11-01T00:17:26.905 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-11-01T00:17:26.905 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rsa-4.9-2.el9.noarch 2023-11-01T00:17:26.905 INFO:teuthology.orchestra.run.smithi072.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-11-01T00:17:26.906 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:26.906 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:17:27.207 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr-cephadm 2023-11-01T00:17:27.749 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:38 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:17:27.891 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:17:27.892 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================== 2023-11-01T00:17:27.892 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2023-11-01T00:17:27.892 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================== 2023-11-01T00:17:27.892 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-11-01T00:17:27.893 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6795.gd1dcabba.el9 ceph-noarch 132 k 2023-11-01T00:17:27.893 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2023-11-01T00:17:27.893 INFO:teuthology.orchestra.run.smithi072.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-11-01T00:17:27.893 INFO:teuthology.orchestra.run.smithi072.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-11-01T00:17:27.894 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-11-01T00:17:27.894 INFO:teuthology.orchestra.run.smithi072.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-11-01T00:17:27.894 INFO:teuthology.orchestra.run.smithi072.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-11-01T00:17:27.894 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:27.895 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-11-01T00:17:27.895 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================== 2023-11-01T00:17:27.895 INFO:teuthology.orchestra.run.smithi072.stdout:Install 6 Packages 2023-11-01T00:17:27.895 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:27.896 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 6.8 M 2023-11-01T00:17:27.896 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 33 M 2023-11-01T00:17:27.896 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-11-01T00:17:28.405 INFO:teuthology.orchestra.run.smithi072.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.5 MB/s | 228 kB 00:00 2023-11-01T00:17:28.556 INFO:teuthology.orchestra.run.smithi072.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2023-11-01T00:17:28.606 INFO:teuthology.orchestra.run.smithi072.stdout:(3/6): ceph-mgr-cephadm-18.0.0-6795.gd1dcabba.e 381 kB/s | 132 kB 00:00 2023-11-01T00:17:28.740 INFO:teuthology.orchestra.run.smithi072.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 313 kB/s | 58 kB 00:00 2023-11-01T00:17:28.765 INFO:teuthology.orchestra.run.smithi072.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 321 kB/s | 51 kB 00:00 2023-11-01T00:17:28.800 INFO:teuthology.orchestra.run.smithi072.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.4 MB/s | 548 kB 00:00 2023-11-01T00:17:28.802 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-11-01T00:17:28.802 INFO:teuthology.orchestra.run.smithi072.stdout:Total 7.4 MB/s | 6.8 MB 00:00 2023-11-01T00:17:28.802 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-11-01T00:17:28.812 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-11-01T00:17:28.812 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-11-01T00:17:28.905 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-11-01T00:17:28.905 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-11-01T00:17:29.204 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-11-01T00:17:29.376 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-11-01T00:17:29.428 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-11-01T00:17:29.953 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-11-01T00:17:30.042 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-11-01T00:17:30.112 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-11-01T00:17:30.146 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6795.gd1dcabba.el9.noarch 6/6 2023-11-01T00:17:31.171 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6795.gd1dcabba.el9.noarch 6/6 2023-11-01T00:17:31.172 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6795.gd1dcabba.el9.noarch 1/6 2023-11-01T00:17:31.172 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-11-01T00:17:31.172 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-11-01T00:17:31.172 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-11-01T00:17:31.172 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-11-01T00:17:31.496 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-11-01T00:17:31.496 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:31.497 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-11-01T00:17:31.497 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-cephadm-2:18.0.0-6795.gd1dcabba.el9.noarch 2023-11-01T00:17:31.497 INFO:teuthology.orchestra.run.smithi072.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-11-01T00:17:31.497 INFO:teuthology.orchestra.run.smithi072.stdout: python3-babel-2.9.1-2.el9.noarch 2023-11-01T00:17:31.497 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-11-01T00:17:31.497 INFO:teuthology.orchestra.run.smithi072.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-11-01T00:17:31.498 INFO:teuthology.orchestra.run.smithi072.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-11-01T00:17:31.498 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:31.498 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:17:31.809 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-fuse 2023-11-01T00:17:32.352 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:43 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:17:32.493 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:17:32.494 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:32.495 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2023-11-01T00:17:32.495 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:32.495 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-11-01T00:17:32.495 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-fuse x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 832 k 2023-11-01T00:17:32.495 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2023-11-01T00:17:32.496 INFO:teuthology.orchestra.run.smithi072.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-11-01T00:17:32.496 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:32.496 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-11-01T00:17:32.496 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:32.496 INFO:teuthology.orchestra.run.smithi072.stdout:Install 2 Packages 2023-11-01T00:17:32.497 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:32.497 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 910 k 2023-11-01T00:17:32.497 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 3.0 M 2023-11-01T00:17:32.497 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-11-01T00:17:32.608 INFO:teuthology.orchestra.run.smithi072.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 699 kB/s | 79 kB 00:00 2023-11-01T00:17:32.834 INFO:teuthology.orchestra.run.smithi072.stdout:(2/2): ceph-fuse-18.0.0-6795.gd1dcabba.el9.x86_ 2.4 MB/s | 832 kB 00:00 2023-11-01T00:17:32.834 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-11-01T00:17:32.835 INFO:teuthology.orchestra.run.smithi072.stdout:Total 2.6 MB/s | 910 kB 00:00 2023-11-01T00:17:32.835 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-11-01T00:17:32.855 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-11-01T00:17:32.855 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-11-01T00:17:32.932 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-11-01T00:17:32.932 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-11-01T00:17:33.107 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-11-01T00:17:33.251 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-11-01T00:17:33.326 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-fuse-2:18.0.0-6795.gd1dcabba.el9.x86_64 2/2 2023-11-01T00:17:34.101 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6795.gd1dcabba.el9.x86_64 2/2 2023-11-01T00:17:34.102 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-fuse-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/2 2023-11-01T00:17:34.434 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-11-01T00:17:34.434 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:34.434 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-11-01T00:17:34.435 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-fuse-2:18.0.0-6795.gd1dcabba.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-11-01T00:17:34.435 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:34.435 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:17:34.783 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-volume 2023-11-01T00:17:35.338 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:46 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:17:35.483 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:17:35.483 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:35.484 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2023-11-01T00:17:35.484 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:35.484 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-11-01T00:17:35.484 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-volume noarch 2:18.0.0-6795.gd1dcabba.el9 ceph-noarch 262 k 2023-11-01T00:17:35.484 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:35.485 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-11-01T00:17:35.485 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:35.485 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2023-11-01T00:17:35.485 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:35.485 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 262 k 2023-11-01T00:17:35.485 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 1.2 M 2023-11-01T00:17:35.486 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-11-01T00:17:35.707 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-volume-18.0.0-6795.gd1dcabba.el9.noarch.rp 1.2 MB/s | 262 kB 00:00 2023-11-01T00:17:35.707 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-11-01T00:17:35.708 INFO:teuthology.orchestra.run.smithi072.stdout:Total 1.1 MB/s | 262 kB 00:00 2023-11-01T00:17:35.708 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-11-01T00:17:35.715 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-11-01T00:17:35.715 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-11-01T00:17:35.749 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-11-01T00:17:35.749 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-11-01T00:17:35.959 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-11-01T00:17:36.027 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-volume-2:18.0.0-6795.gd1dcabba.el9.noarch 1/1 2023-11-01T00:17:36.042 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-volume-2:18.0.0-6795.gd1dcabba.el9.noarch 1/1 2023-11-01T00:17:36.042 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-01T00:17:36.042 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-11-01T00:17:36.043 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:37.111 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-volume-2:18.0.0-6795.gd1dcabba.el9.noarch 1/1 2023-11-01T00:17:37.112 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:37.112 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-11-01T00:17:37.112 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-volume-2:18.0.0-6795.gd1dcabba.el9.noarch 2023-11-01T00:17:37.112 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:37.113 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:17:37.494 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install librados-devel 2023-11-01T00:17:38.051 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:49 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:17:38.194 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:17:38.195 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:38.195 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repo Size 2023-11-01T00:17:38.195 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:38.196 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-11-01T00:17:38.196 INFO:teuthology.orchestra.run.smithi072.stdout: librados-devel x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 127 k 2023-11-01T00:17:38.196 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:38.196 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-11-01T00:17:38.196 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:38.196 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2023-11-01T00:17:38.197 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:38.197 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 127 k 2023-11-01T00:17:38.197 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 456 k 2023-11-01T00:17:38.197 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-11-01T00:17:38.414 INFO:teuthology.orchestra.run.smithi072.stdout:librados-devel-18.0.0-6795.gd1dcabba.el9.x86_64 581 kB/s | 127 kB 00:00 2023-11-01T00:17:38.415 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-11-01T00:17:38.415 INFO:teuthology.orchestra.run.smithi072.stdout:Total 578 kB/s | 127 kB 00:00 2023-11-01T00:17:38.415 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-11-01T00:17:38.420 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-11-01T00:17:38.420 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-11-01T00:17:38.474 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-11-01T00:17:38.475 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-11-01T00:17:38.612 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-11-01T00:17:38.721 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librados-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-11-01T00:17:39.266 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-11-01T00:17:39.568 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-11-01T00:17:39.569 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:39.569 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-11-01T00:17:39.569 INFO:teuthology.orchestra.run.smithi072.stdout: librados-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:17:39.569 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:39.570 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:17:39.896 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install libcephfs2 2023-11-01T00:17:40.440 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:51 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:17:40.533 INFO:teuthology.orchestra.run.smithi072.stdout:Package libcephfs2-2:18.0.0-6795.gd1dcabba.el9.x86_64 is already installed. 2023-11-01T00:17:40.582 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:17:40.583 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2023-11-01T00:17:40.583 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:17:40.668 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install libcephfs-devel 2023-11-01T00:17:41.212 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:52 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:17:41.352 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:17:41.353 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:41.353 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repo Size 2023-11-01T00:17:41.354 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:41.354 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-11-01T00:17:41.354 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs-devel x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 31 k 2023-11-01T00:17:41.354 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:41.355 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-11-01T00:17:41.355 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:41.355 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2023-11-01T00:17:41.355 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:41.355 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 31 k 2023-11-01T00:17:41.355 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 139 k 2023-11-01T00:17:41.356 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-11-01T00:17:41.860 INFO:teuthology.orchestra.run.smithi072.stdout:libcephfs-devel-18.0.0-6795.gd1dcabba.el9.x86_6 61 kB/s | 31 kB 00:00 2023-11-01T00:17:41.861 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-11-01T00:17:41.861 INFO:teuthology.orchestra.run.smithi072.stdout:Total 61 kB/s | 31 kB 00:00 2023-11-01T00:17:41.861 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-11-01T00:17:41.864 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-11-01T00:17:41.864 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-11-01T00:17:41.899 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-11-01T00:17:41.899 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-11-01T00:17:41.989 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-11-01T00:17:42.109 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephfs-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-11-01T00:17:42.535 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-11-01T00:17:42.809 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephfs-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-11-01T00:17:42.810 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:42.810 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-11-01T00:17:42.810 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:17:42.810 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:42.811 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:17:43.023 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install librados2 2023-11-01T00:17:43.564 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:54 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:17:43.655 INFO:teuthology.orchestra.run.smithi072.stdout:Package librados2-2:18.0.0-6795.gd1dcabba.el9.x86_64 is already installed. 2023-11-01T00:17:43.705 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:17:43.706 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2023-11-01T00:17:43.706 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:17:43.799 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install librbd1 2023-11-01T00:17:44.342 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:55 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:17:44.433 INFO:teuthology.orchestra.run.smithi072.stdout:Package librbd1-2:18.0.0-6795.gd1dcabba.el9.x86_64 is already installed. 2023-11-01T00:17:44.482 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:17:44.483 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2023-11-01T00:17:44.483 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:17:44.573 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install python3-rados 2023-11-01T00:17:45.113 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:56 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:17:45.202 INFO:teuthology.orchestra.run.smithi072.stdout:Package python3-rados-2:18.0.0-6795.gd1dcabba.el9.x86_64 is already installed. 2023-11-01T00:17:45.252 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:17:45.253 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2023-11-01T00:17:45.253 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:17:45.340 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install python3-rgw 2023-11-01T00:17:45.881 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:56 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:17:45.972 INFO:teuthology.orchestra.run.smithi072.stdout:Package python3-rgw-2:18.0.0-6795.gd1dcabba.el9.x86_64 is already installed. 2023-11-01T00:17:46.021 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:17:46.022 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2023-11-01T00:17:46.022 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:17:46.114 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install python3-cephfs 2023-11-01T00:17:46.657 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:57 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:17:46.747 INFO:teuthology.orchestra.run.smithi072.stdout:Package python3-cephfs-2:18.0.0-6795.gd1dcabba.el9.x86_64 is already installed. 2023-11-01T00:17:46.796 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:17:46.797 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2023-11-01T00:17:46.797 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:17:46.887 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install python3-rbd 2023-11-01T00:17:47.429 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:58 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:17:47.519 INFO:teuthology.orchestra.run.smithi072.stdout:Package python3-rbd-2:18.0.0-6795.gd1dcabba.el9.x86_64 is already installed. 2023-11-01T00:17:47.568 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:17:47.569 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2023-11-01T00:17:47.569 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:17:47.656 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install rbd-fuse 2023-11-01T00:17:48.197 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:59 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:17:48.339 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:17:48.340 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:48.340 INFO:teuthology.orchestra.run.smithi072.stdout: Package Architecture Version Repository Size 2023-11-01T00:17:48.340 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:48.340 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-11-01T00:17:48.340 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-fuse x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 87 k 2023-11-01T00:17:48.341 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:48.341 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-11-01T00:17:48.341 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:48.341 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2023-11-01T00:17:48.341 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:48.342 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 87 k 2023-11-01T00:17:48.342 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 252 k 2023-11-01T00:17:48.342 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-11-01T00:17:48.785 INFO:teuthology.orchestra.run.smithi072.stdout:rbd-fuse-18.0.0-6795.gd1dcabba.el9.x86_64.rpm 195 kB/s | 87 kB 00:00 2023-11-01T00:17:48.785 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-11-01T00:17:48.786 INFO:teuthology.orchestra.run.smithi072.stdout:Total 194 kB/s | 87 kB 00:00 2023-11-01T00:17:48.786 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-11-01T00:17:48.792 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-11-01T00:17:48.793 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-11-01T00:17:48.848 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-11-01T00:17:48.849 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-11-01T00:17:48.983 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-11-01T00:17:49.083 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-fuse-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-11-01T00:17:49.534 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-11-01T00:17:49.873 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-fuse-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-11-01T00:17:49.873 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:49.873 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-11-01T00:17:49.874 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-fuse-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:17:49.874 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:49.874 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:17:50.163 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install rbd-mirror 2023-11-01T00:17:50.710 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:03:01 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:17:50.857 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:17:50.858 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:50.859 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repo Size 2023-11-01T00:17:50.859 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:50.859 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-11-01T00:17:50.859 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-mirror x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 3.0 M 2023-11-01T00:17:50.859 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:50.860 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-11-01T00:17:50.860 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:50.860 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2023-11-01T00:17:50.860 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:50.860 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 3.0 M 2023-11-01T00:17:50.861 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 13 M 2023-11-01T00:17:50.861 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-11-01T00:17:51.255 INFO:teuthology.orchestra.run.smithi072.stdout:rbd-mirror-18.0.0-6795.gd1dcabba.el9.x86_64.rpm 7.6 MB/s | 3.0 MB 00:00 2023-11-01T00:17:51.255 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-11-01T00:17:51.256 INFO:teuthology.orchestra.run.smithi072.stdout:Total 7.6 MB/s | 3.0 MB 00:00 2023-11-01T00:17:51.256 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-11-01T00:17:51.264 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-11-01T00:17:51.264 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-11-01T00:17:51.343 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-11-01T00:17:51.343 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-11-01T00:17:51.759 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-11-01T00:17:51.856 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-mirror-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-11-01T00:17:51.880 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-11-01T00:17:51.880 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-11-01T00:17:51.881 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-11-01T00:17:51.881 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-01T00:17:51.881 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-11-01T00:17:51.881 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:53.229 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-mirror-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-11-01T00:17:53.230 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:53.230 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-11-01T00:17:53.230 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-mirror-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:17:53.230 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:53.231 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:17:53.533 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install rbd-nbd 2023-11-01T00:17:54.081 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:03:05 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:17:54.222 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:17:54.223 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:54.223 INFO:teuthology.orchestra.run.smithi072.stdout: Package Architecture Version Repository Size 2023-11-01T00:17:54.223 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:54.224 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-11-01T00:17:54.224 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-nbd x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 168 k 2023-11-01T00:17:54.224 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:54.224 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-11-01T00:17:54.224 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:17:54.225 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2023-11-01T00:17:54.225 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:54.225 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 168 k 2023-11-01T00:17:54.225 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 532 k 2023-11-01T00:17:54.225 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-11-01T00:17:54.458 INFO:teuthology.orchestra.run.smithi072.stdout:rbd-nbd-18.0.0-6795.gd1dcabba.el9.x86_64.rpm 720 kB/s | 168 kB 00:00 2023-11-01T00:17:54.459 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-11-01T00:17:54.459 INFO:teuthology.orchestra.run.smithi072.stdout:Total 715 kB/s | 168 kB 00:00 2023-11-01T00:17:54.459 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-11-01T00:17:54.466 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-11-01T00:17:54.467 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-11-01T00:17:54.522 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-11-01T00:17:54.523 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-11-01T00:17:54.676 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-11-01T00:17:54.782 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-nbd-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-11-01T00:17:55.266 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-11-01T00:17:55.580 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-nbd-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/1 2023-11-01T00:17:55.580 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:55.581 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-11-01T00:17:55.581 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-nbd-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:17:55.581 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:55.581 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:17:55.883 DEBUG:teuthology.parallel:result is None 2023-11-01T00:17:55.883 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=d1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2 2023-11-01T00:17:56.051 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-01T00:17:56.072 INFO:teuthology.orchestra.run.smithi072.stdout:18.0.0-6795.gd1dcabba.el9 2023-11-01T00:17:56.073 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6795.gd1dcabba.el9 2023-11-01T00:17:56.073 INFO:teuthology.task.install:The correct ceph version 18.0.0-6795.gd1dcabba is installed. 2023-11-01T00:17:56.074 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-01T00:17:56.075 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-11-01T00:17:56.076 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-01T00:17:56.139 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-01T00:17:56.140 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-11-01T00:17:56.140 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/daemon-helper 2023-11-01T00:17:56.206 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-01T00:17:56.272 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-01T00:17:56.272 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-11-01T00:17:56.273 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-01T00:17:56.338 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-01T00:17:56.405 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-01T00:17:56.406 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-11-01T00:17:56.406 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/stdin-killer 2023-11-01T00:17:56.472 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-01T00:17:56.538 INFO:teuthology.run_tasks:Running task ceph... 2023-11-01T00:17:56.600 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-11-01T00:17:56.601 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 777 /var/log/ceph 2023-11-01T00:17:56.627 INFO:tasks.ceph:Disabling ceph logrotate... 2023-11-01T00:17:56.628 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /etc/logrotate.d/ceph 2023-11-01T00:17:56.694 INFO:tasks.ceph:Creating extra log directories... 2023-11-01T00:17:56.695 DEBUG:teuthology.orchestra.run.smithi072:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-11-01T00:17:56.768 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-11-01T00:17:56.769 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-11-01T00:17:56.769 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi/7442572', 'branch': 'wip-matanb-crimson-do_osd_ops_execute-part3', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write}', 'email': None, 'first_in_suite': False, 'job_id': '7442572', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-matanb-crimson-do_osd_ops_execute-part3'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'd1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': 'd1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2'}}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'a9edd23b5a8ddacab225824948b06b370baf276b'}}, 'owner': 'scheduled_matan@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 6029, 'sha1': 'd1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_a9edd23b5a8ddacab225824948b06b370baf276b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': 'a9edd23b5a8ddacab225824948b06b370baf276b', 'targets': {'smithi072.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPqQ8m1lyC4gfmIhCRK9ue+TQpRlEZDrwDAYyVhYmeVhgru8v0m8+bPoyhVsLbnuMQxri2+3S78DUdW7KMv+nhg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'd1dcabbaf4ab585825bf6f16fc9df7bfd64a85b2', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-10-31_22:58:07', 'tube': 'smithi', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.770726'} 2023-11-01T00:17:56.770 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-11-01T00:17:56.816 DEBUG:teuthology.orchestra.run.smithi072:> sudo install -d -m0777 -- /var/run/ceph 2023-11-01T00:17:56.883 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-11-01T00:17:56.883 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/scratch_devs of=/dev/stdout 2023-11-01T00:17:56.937 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-01T00:17:56.938 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_1 2023-11-01T00:17:56.992 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-01T00:17:56.992 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-01T00:17:56.992 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 663 Links: 1 2023-11-01T00:17:56.992 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-01T00:17:56.993 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2023-11-01T00:17:56.993 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2023-11-01 00:17:52.129318537 +0000 2023-11-01T00:17:56.993 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2023-11-01 00:14:04.841507962 +0000 2023-11-01T00:17:56.993 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2023-11-01 00:14:04.841507962 +0000 2023-11-01T00:17:56.993 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2023-11-01 00:14:04.841507962 +0000 2023-11-01T00:17:56.994 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-01T00:17:57.057 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2023-11-01T00:17:57.057 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2023-11-01T00:17:57.057 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000177973 s, 2.9 MB/s 2023-11-01T00:17:57.058 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-01T00:17:57.113 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_2 2023-11-01T00:17:57.168 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-01T00:17:57.168 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-01T00:17:57.168 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 680 Links: 1 2023-11-01T00:17:57.168 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-01T00:17:57.168 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2023-11-01T00:17:57.169 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2023-11-01 00:17:52.129318537 +0000 2023-11-01T00:17:57.169 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2023-11-01 00:14:05.329497309 +0000 2023-11-01T00:17:57.169 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2023-11-01 00:14:05.329497309 +0000 2023-11-01T00:17:57.169 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2023-11-01 00:14:05.329497309 +0000 2023-11-01T00:17:57.169 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-01T00:17:57.233 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2023-11-01T00:17:57.233 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2023-11-01T00:17:57.233 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000202744 s, 2.5 MB/s 2023-11-01T00:17:57.234 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-01T00:17:57.290 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_3 2023-11-01T00:17:57.345 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-01T00:17:57.346 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-01T00:17:57.346 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 698 Links: 1 2023-11-01T00:17:57.346 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-01T00:17:57.346 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2023-11-01T00:17:57.346 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2023-11-01 00:17:52.130318515 +0000 2023-11-01T00:17:57.347 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2023-11-01 00:14:05.815486699 +0000 2023-11-01T00:17:57.347 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2023-11-01 00:14:05.815486699 +0000 2023-11-01T00:17:57.347 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2023-11-01 00:14:05.815486699 +0000 2023-11-01T00:17:57.347 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-01T00:17:57.412 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2023-11-01T00:17:57.412 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2023-11-01T00:17:57.412 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000186214 s, 2.7 MB/s 2023-11-01T00:17:57.413 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-01T00:17:57.468 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_4 2023-11-01T00:17:57.523 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-01T00:17:57.523 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-01T00:17:57.523 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 718 Links: 1 2023-11-01T00:17:57.523 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-01T00:17:57.523 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2023-11-01T00:17:57.524 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2023-11-01 00:17:52.130318515 +0000 2023-11-01T00:17:57.524 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2023-11-01 00:14:06.289476351 +0000 2023-11-01T00:17:57.524 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2023-11-01 00:14:06.289476351 +0000 2023-11-01T00:17:57.524 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2023-11-01 00:14:06.289476351 +0000 2023-11-01T00:17:57.525 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-01T00:17:57.588 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2023-11-01T00:17:57.589 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2023-11-01T00:17:57.589 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000185008 s, 2.8 MB/s 2023-11-01T00:17:57.590 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-01T00:17:57.645 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-01T00:17:57.645 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi072.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-01T00:17:57.645 INFO:tasks.ceph:Generating config... 2023-11-01T00:17:57.647 INFO:tasks.ceph:[osd] debug monc = 20 2023-11-01T00:17:57.647 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-11-01T00:17:57.647 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2023-11-01T00:17:57.648 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-11-01T00:17:57.648 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-11-01T00:17:57.648 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2023-11-01T00:17:57.648 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2023-11-01T00:17:57.649 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-11-01T00:17:57.649 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-11-01T00:17:57.649 INFO:tasks.ceph:[osd] debug filestore = 0/0 2023-11-01T00:17:57.650 INFO:tasks.ceph:[osd] debug journal = 0/0 2023-11-01T00:17:57.650 INFO:tasks.ceph:[osd] debug ms = 0/0 2023-11-01T00:17:57.650 INFO:tasks.ceph:[osd] debug osd = 0/0 2023-11-01T00:17:57.650 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-11-01T00:17:57.651 INFO:tasks.ceph:[osd] err to stderr = True 2023-11-01T00:17:57.651 INFO:tasks.ceph:[osd] log flush on exit = True 2023-11-01T00:17:57.651 INFO:tasks.ceph:[osd] log to file = False 2023-11-01T00:17:57.651 INFO:tasks.ceph:[osd] log to stderr = True 2023-11-01T00:17:57.652 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-11-01T00:17:57.652 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-11-01T00:17:57.652 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-11-01T00:17:57.652 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-11-01T00:17:57.653 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-11-01T00:17:57.653 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-11-01T00:17:57.653 INFO:tasks.ceph:[global] auth client required = none 2023-11-01T00:17:57.653 INFO:tasks.ceph:[global] auth cluster required = none 2023-11-01T00:17:57.654 INFO:tasks.ceph:[global] auth service required = none 2023-11-01T00:17:57.654 INFO:tasks.ceph:[global] auth supported = none 2023-11-01T00:17:57.654 INFO:tasks.ceph:[global] debug asok = 0/0 2023-11-01T00:17:57.654 INFO:tasks.ceph:[global] debug auth = 0/0 2023-11-01T00:17:57.655 INFO:tasks.ceph:[global] debug bdev = 0/0 2023-11-01T00:17:57.655 INFO:tasks.ceph:[global] debug bluefs = 0/0 2023-11-01T00:17:57.655 INFO:tasks.ceph:[global] debug bluestore = 0/0 2023-11-01T00:17:57.656 INFO:tasks.ceph:[global] debug buffer = 0/0 2023-11-01T00:17:57.656 INFO:tasks.ceph:[global] debug civetweb = 0/0 2023-11-01T00:17:57.656 INFO:tasks.ceph:[global] debug client = 0/0 2023-11-01T00:17:57.656 INFO:tasks.ceph:[global] debug compressor = 0/0 2023-11-01T00:17:57.657 INFO:tasks.ceph:[global] debug context = 0/0 2023-11-01T00:17:57.657 INFO:tasks.ceph:[global] debug crush = 0/0 2023-11-01T00:17:57.657 INFO:tasks.ceph:[global] debug crypto = 0/0 2023-11-01T00:17:57.657 INFO:tasks.ceph:[global] debug dpdk = 0/0 2023-11-01T00:17:57.658 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2023-11-01T00:17:57.658 INFO:tasks.ceph:[global] debug filer = 0/0 2023-11-01T00:17:57.658 INFO:tasks.ceph:[global] debug filestore = 0/0 2023-11-01T00:17:57.658 INFO:tasks.ceph:[global] debug finisher = 0/0 2023-11-01T00:17:57.659 INFO:tasks.ceph:[global] debug fuse = 0/0 2023-11-01T00:17:57.659 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2023-11-01T00:17:57.659 INFO:tasks.ceph:[global] debug javaclient = 0/0 2023-11-01T00:17:57.659 INFO:tasks.ceph:[global] debug journal = 0/0 2023-11-01T00:17:57.660 INFO:tasks.ceph:[global] debug journaler = 0/0 2023-11-01T00:17:57.660 INFO:tasks.ceph:[global] debug kstore = 0/0 2023-11-01T00:17:57.660 INFO:tasks.ceph:[global] debug leveldb = 0/0 2023-11-01T00:17:57.660 INFO:tasks.ceph:[global] debug lockdep = 0/0 2023-11-01T00:17:57.660 INFO:tasks.ceph:[global] debug mds = 0/0 2023-11-01T00:17:57.661 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2023-11-01T00:17:57.661 INFO:tasks.ceph:[global] debug mds locker = 0/0 2023-11-01T00:17:57.661 INFO:tasks.ceph:[global] debug mds log = 0/0 2023-11-01T00:17:57.661 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2023-11-01T00:17:57.662 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2023-11-01T00:17:57.662 INFO:tasks.ceph:[global] debug memdb = 0/0 2023-11-01T00:17:57.662 INFO:tasks.ceph:[global] debug mgr = 0/0 2023-11-01T00:17:57.663 INFO:tasks.ceph:[global] debug mgrc = 0/0 2023-11-01T00:17:57.663 INFO:tasks.ceph:[global] debug mon = 0/0 2023-11-01T00:17:57.663 INFO:tasks.ceph:[global] debug monc = 0/0 2023-11-01T00:17:57.663 INFO:tasks.ceph:[global] debug ms = 0/0 2023-11-01T00:17:57.664 INFO:tasks.ceph:[global] debug objclass = 0/0 2023-11-01T00:17:57.664 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2023-11-01T00:17:57.664 INFO:tasks.ceph:[global] debug objecter = 0/0 2023-11-01T00:17:57.664 INFO:tasks.ceph:[global] debug optracker = 0/0 2023-11-01T00:17:57.665 INFO:tasks.ceph:[global] debug osd = 0/0 2023-11-01T00:17:57.665 INFO:tasks.ceph:[global] debug paxos = 0/0 2023-11-01T00:17:57.665 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2023-11-01T00:17:57.665 INFO:tasks.ceph:[global] debug rados = 0/0 2023-11-01T00:17:57.666 INFO:tasks.ceph:[global] debug rbd = 0/0 2023-11-01T00:17:57.666 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2023-11-01T00:17:57.666 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2023-11-01T00:17:57.667 INFO:tasks.ceph:[global] debug refs = 0/0 2023-11-01T00:17:57.667 INFO:tasks.ceph:[global] debug rgw = 0/0 2023-11-01T00:17:57.667 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2023-11-01T00:17:57.667 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2023-11-01T00:17:57.668 INFO:tasks.ceph:[global] debug striper = 0/0 2023-11-01T00:17:57.668 INFO:tasks.ceph:[global] debug throttle = 0/0 2023-11-01T00:17:57.668 INFO:tasks.ceph:[global] debug timer = 0/0 2023-11-01T00:17:57.668 INFO:tasks.ceph:[global] debug tp = 0/0 2023-11-01T00:17:57.669 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2023-11-01T00:17:57.669 INFO:tasks.ceph:[global] ms client mode = crc 2023-11-01T00:17:57.669 INFO:tasks.ceph:[global] ms cluster mode = crc 2023-11-01T00:17:57.669 INFO:tasks.ceph:[global] ms mon client mode = crc 2023-11-01T00:17:57.669 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2023-11-01T00:17:57.670 INFO:tasks.ceph:[global] ms mon service mode = crc 2023-11-01T00:17:57.670 INFO:tasks.ceph:[global] ms service mode = crc 2023-11-01T00:17:57.670 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-11-01T00:17:57.670 INFO:tasks.ceph:[mgr] debug ms = 1 2023-11-01T00:17:57.671 INFO:tasks.ceph:[mon] debug mon = 0/0 2023-11-01T00:17:57.671 INFO:tasks.ceph:[mon] debug ms = 0/0 2023-11-01T00:17:57.671 INFO:tasks.ceph:[mon] debug paxos = 0/0 2023-11-01T00:17:57.671 INFO:tasks.ceph:[mon] osd pool default crimson = True 2023-11-01T00:17:57.672 INFO:tasks.ceph:Setting up mon.a... 2023-11-01T00:17:57.672 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-11-01T00:17:57.736 INFO:teuthology.orchestra.run.smithi072.stdout:creating /etc/ceph/ceph.keyring 2023-11-01T00:17:57.739 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-11-01T00:17:57.788 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-01T00:17:57.814 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.72')] 2023-11-01T00:17:57.815 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.72', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2023-11-01T00:17:57.816 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-11-01T00:17:57.816 DEBUG:teuthology.orchestra.run.smithi072:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-11-01T00:17:57.868 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.72 --print /home/ubuntu/cephtest/ceph.monmap 2023-11-01T00:17:57.952 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:17:57.951+0000 7f1c0cc46e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:17:57.952 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:17:57.951+0000 7f1c0cc46e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:17:57.953 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:17:57.951+0000 7f1c0cc46e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:17:57.954 INFO:teuthology.orchestra.run.smithi072.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-11-01T00:17:57.954 INFO:teuthology.orchestra.run.smithi072.stdout:monmaptool: generated fsid ef82487d-963f-4ecc-9718-b8d0ac62573e 2023-11-01T00:17:57.954 INFO:teuthology.orchestra.run.smithi072.stdout:setting min_mon_release = pacific 2023-11-01T00:17:57.954 INFO:teuthology.orchestra.run.smithi072.stdout:epoch 0 2023-11-01T00:17:57.955 INFO:teuthology.orchestra.run.smithi072.stdout:fsid ef82487d-963f-4ecc-9718-b8d0ac62573e 2023-11-01T00:17:57.955 INFO:teuthology.orchestra.run.smithi072.stdout:last_changed 2023-11-01T00:17:57.952485+0000 2023-11-01T00:17:57.955 INFO:teuthology.orchestra.run.smithi072.stdout:created 2023-11-01T00:17:57.952485+0000 2023-11-01T00:17:57.955 INFO:teuthology.orchestra.run.smithi072.stdout:min_mon_release 16 (pacific) 2023-11-01T00:17:57.955 INFO:teuthology.orchestra.run.smithi072.stdout:election_strategy: 1 2023-11-01T00:17:57.955 INFO:teuthology.orchestra.run.smithi072.stdout:0: [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon.a 2023-11-01T00:17:57.956 INFO:teuthology.orchestra.run.smithi072.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-11-01T00:17:57.956 DEBUG:teuthology.orchestra.run.smithi072:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-11-01T00:17:57.970 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ef82487d-963f-4ecc-9718-b8d0ac62573e... 2023-11-01T00:17:57.971 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-11-01T00:17:58.060 INFO:teuthology.orchestra.run.smithi072.stdout:[global] 2023-11-01T00:17:58.060 INFO:teuthology.orchestra.run.smithi072.stdout: chdir = "" 2023-11-01T00:17:58.060 INFO:teuthology.orchestra.run.smithi072.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-11-01T00:17:58.061 INFO:teuthology.orchestra.run.smithi072.stdout: auth supported = none 2023-11-01T00:17:58.061 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.061 INFO:teuthology.orchestra.run.smithi072.stdout: filestore xattr use omap = true 2023-11-01T00:17:58.061 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.061 INFO:teuthology.orchestra.run.smithi072.stdout: mon clock drift allowed = 1.000 2023-11-01T00:17:58.061 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.062 INFO:teuthology.orchestra.run.smithi072.stdout: osd crush chooseleaf type = 0 2023-11-01T00:17:58.062 INFO:teuthology.orchestra.run.smithi072.stdout: auth debug = true 2023-11-01T00:17:58.062 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.062 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on old message = true 2023-11-01T00:17:58.062 INFO:teuthology.orchestra.run.smithi072.stdout: ms die on bug = true 2023-11-01T00:17:58.063 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.063 INFO:teuthology.orchestra.run.smithi072.stdout: mon max pg per osd = 10000 # >= luminous 2023-11-01T00:17:58.063 INFO:teuthology.orchestra.run.smithi072.stdout: mon pg warn max object skew = 0 2023-11-01T00:17:58.063 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.063 INFO:teuthology.orchestra.run.smithi072.stdout: # disable pg_autoscaler by default for new pools 2023-11-01T00:17:58.064 INFO:teuthology.orchestra.run.smithi072.stdout: osd_pool_default_pg_autoscale_mode = off 2023-11-01T00:17:58.064 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.064 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default size = 2 2023-11-01T00:17:58.064 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.064 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow primary affinity = true 2023-11-01T00:17:58.064 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd allow pg remap = true 2023-11-01T00:17:58.065 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on legacy crush tunables = false 2023-11-01T00:17:58.065 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on crush straw calc version zero = false 2023-11-01T00:17:58.065 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on no sortbitwise = false 2023-11-01T00:17:58.065 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on osd down out interval zero = false 2023-11-01T00:17:58.065 INFO:teuthology.orchestra.run.smithi072.stdout: mon warn on too few osds = false 2023-11-01T00:17:58.066 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-11-01T00:17:58.066 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_pool_no_redundancy = false 2023-11-01T00:17:58.066 INFO:teuthology.orchestra.run.smithi072.stdout: mon_allow_pool_size_one = true 2023-11-01T00:17:58.066 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.066 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-11-01T00:17:58.067 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.067 INFO:teuthology.orchestra.run.smithi072.stdout: osd default data pool replay window = 5 2023-11-01T00:17:58.067 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.067 INFO:teuthology.orchestra.run.smithi072.stdout: mon allow pool delete = true 2023-11-01T00:17:58.067 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.067 INFO:teuthology.orchestra.run.smithi072.stdout: mon cluster log file level = debug 2023-11-01T00:17:58.068 INFO:teuthology.orchestra.run.smithi072.stdout: debug asserts on shutdown = true 2023-11-01T00:17:58.068 INFO:teuthology.orchestra.run.smithi072.stdout: mon health detail to clog = false 2023-11-01T00:17:58.068 INFO:teuthology.orchestra.run.smithi072.stdout: mon host = 172.21.15.72 2023-11-01T00:17:58.068 INFO:teuthology.orchestra.run.smithi072.stdout: auth client required = none 2023-11-01T00:17:58.068 INFO:teuthology.orchestra.run.smithi072.stdout: auth cluster required = none 2023-11-01T00:17:58.069 INFO:teuthology.orchestra.run.smithi072.stdout: auth service required = none 2023-11-01T00:17:58.069 INFO:teuthology.orchestra.run.smithi072.stdout: debug asok = 0/0 2023-11-01T00:17:58.069 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 0/0 2023-11-01T00:17:58.069 INFO:teuthology.orchestra.run.smithi072.stdout: debug bdev = 0/0 2023-11-01T00:17:58.069 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluefs = 0/0 2023-11-01T00:17:58.069 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluestore = 0/0 2023-11-01T00:17:58.070 INFO:teuthology.orchestra.run.smithi072.stdout: debug buffer = 0/0 2023-11-01T00:17:58.070 INFO:teuthology.orchestra.run.smithi072.stdout: debug civetweb = 0/0 2023-11-01T00:17:58.070 INFO:teuthology.orchestra.run.smithi072.stdout: debug client = 0/0 2023-11-01T00:17:58.070 INFO:teuthology.orchestra.run.smithi072.stdout: debug compressor = 0/0 2023-11-01T00:17:58.070 INFO:teuthology.orchestra.run.smithi072.stdout: debug context = 0/0 2023-11-01T00:17:58.071 INFO:teuthology.orchestra.run.smithi072.stdout: debug crush = 0/0 2023-11-01T00:17:58.071 INFO:teuthology.orchestra.run.smithi072.stdout: debug crypto = 0/0 2023-11-01T00:17:58.071 INFO:teuthology.orchestra.run.smithi072.stdout: debug dpdk = 0/0 2023-11-01T00:17:58.071 INFO:teuthology.orchestra.run.smithi072.stdout: debug eventtrace = 0/0 2023-11-01T00:17:58.071 INFO:teuthology.orchestra.run.smithi072.stdout: debug filer = 0/0 2023-11-01T00:17:58.071 INFO:teuthology.orchestra.run.smithi072.stdout: debug filestore = 0/0 2023-11-01T00:17:58.072 INFO:teuthology.orchestra.run.smithi072.stdout: debug finisher = 0/0 2023-11-01T00:17:58.072 INFO:teuthology.orchestra.run.smithi072.stdout: debug fuse = 0/0 2023-11-01T00:17:58.072 INFO:teuthology.orchestra.run.smithi072.stdout: debug heartbeatmap = 0/0 2023-11-01T00:17:58.072 INFO:teuthology.orchestra.run.smithi072.stdout: debug javaclient = 0/0 2023-11-01T00:17:58.072 INFO:teuthology.orchestra.run.smithi072.stdout: debug journal = 0/0 2023-11-01T00:17:58.073 INFO:teuthology.orchestra.run.smithi072.stdout: debug journaler = 0/0 2023-11-01T00:17:58.073 INFO:teuthology.orchestra.run.smithi072.stdout: debug kstore = 0/0 2023-11-01T00:17:58.073 INFO:teuthology.orchestra.run.smithi072.stdout: debug leveldb = 0/0 2023-11-01T00:17:58.073 INFO:teuthology.orchestra.run.smithi072.stdout: debug lockdep = 0/0 2023-11-01T00:17:58.073 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds = 0/0 2023-11-01T00:17:58.073 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds balancer = 0/0 2023-11-01T00:17:58.074 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds locker = 0/0 2023-11-01T00:17:58.074 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds log = 0/0 2023-11-01T00:17:58.074 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds log expire = 0/0 2023-11-01T00:17:58.074 INFO:teuthology.orchestra.run.smithi072.stdout: debug mds migrator = 0/0 2023-11-01T00:17:58.074 INFO:teuthology.orchestra.run.smithi072.stdout: debug memdb = 0/0 2023-11-01T00:17:58.075 INFO:teuthology.orchestra.run.smithi072.stdout: debug mgr = 0/0 2023-11-01T00:17:58.075 INFO:teuthology.orchestra.run.smithi072.stdout: debug mgrc = 0/0 2023-11-01T00:17:58.075 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 0/0 2023-11-01T00:17:58.075 INFO:teuthology.orchestra.run.smithi072.stdout: debug monc = 0/0 2023-11-01T00:17:58.075 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 0/0 2023-11-01T00:17:58.075 INFO:teuthology.orchestra.run.smithi072.stdout: debug objclass = 0/0 2023-11-01T00:17:58.076 INFO:teuthology.orchestra.run.smithi072.stdout: debug objectcacher = 0/0 2023-11-01T00:17:58.076 INFO:teuthology.orchestra.run.smithi072.stdout: debug objecter = 0/0 2023-11-01T00:17:58.076 INFO:teuthology.orchestra.run.smithi072.stdout: debug optracker = 0/0 2023-11-01T00:17:58.076 INFO:teuthology.orchestra.run.smithi072.stdout: debug osd = 0/0 2023-11-01T00:17:58.076 INFO:teuthology.orchestra.run.smithi072.stdout: debug paxos = 0/0 2023-11-01T00:17:58.077 INFO:teuthology.orchestra.run.smithi072.stdout: debug perfcounter = 0/0 2023-11-01T00:17:58.077 INFO:teuthology.orchestra.run.smithi072.stdout: debug rados = 0/0 2023-11-01T00:17:58.077 INFO:teuthology.orchestra.run.smithi072.stdout: debug rbd = 0/0 2023-11-01T00:17:58.077 INFO:teuthology.orchestra.run.smithi072.stdout: debug rbd mirror = 0/0 2023-11-01T00:17:58.077 INFO:teuthology.orchestra.run.smithi072.stdout: debug rbd replay = 0/0 2023-11-01T00:17:58.077 INFO:teuthology.orchestra.run.smithi072.stdout: debug refs = 0/0 2023-11-01T00:17:58.078 INFO:teuthology.orchestra.run.smithi072.stdout: debug rgw = 0/0 2023-11-01T00:17:58.078 INFO:teuthology.orchestra.run.smithi072.stdout: debug rgw sync = 0/0 2023-11-01T00:17:58.078 INFO:teuthology.orchestra.run.smithi072.stdout: debug rocksdb = 0/0 2023-11-01T00:17:58.078 INFO:teuthology.orchestra.run.smithi072.stdout: debug striper = 0/0 2023-11-01T00:17:58.078 INFO:teuthology.orchestra.run.smithi072.stdout: debug throttle = 0/0 2023-11-01T00:17:58.078 INFO:teuthology.orchestra.run.smithi072.stdout: debug timer = 0/0 2023-11-01T00:17:58.079 INFO:teuthology.orchestra.run.smithi072.stdout: debug tp = 0/0 2023-11-01T00:17:58.079 INFO:teuthology.orchestra.run.smithi072.stdout: enable experimental unrecoverable data corrupting features = crimson 2023-11-01T00:17:58.079 INFO:teuthology.orchestra.run.smithi072.stdout: ms client mode = crc 2023-11-01T00:17:58.079 INFO:teuthology.orchestra.run.smithi072.stdout: ms cluster mode = crc 2023-11-01T00:17:58.079 INFO:teuthology.orchestra.run.smithi072.stdout: ms mon client mode = crc 2023-11-01T00:17:58.079 INFO:teuthology.orchestra.run.smithi072.stdout: ms mon cluster mode = crc 2023-11-01T00:17:58.080 INFO:teuthology.orchestra.run.smithi072.stdout: ms mon service mode = crc 2023-11-01T00:17:58.080 INFO:teuthology.orchestra.run.smithi072.stdout: ms service mode = crc 2023-11-01T00:17:58.080 INFO:teuthology.orchestra.run.smithi072.stdout: fsid = ef82487d-963f-4ecc-9718-b8d0ac62573e 2023-11-01T00:17:58.080 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.080 INFO:teuthology.orchestra.run.smithi072.stdout:[osd] 2023-11-01T00:17:58.081 INFO:teuthology.orchestra.run.smithi072.stdout: osd journal size = 100 2023-11-01T00:17:58.081 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.081 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub load threshold = 5.0 2023-11-01T00:17:58.081 INFO:teuthology.orchestra.run.smithi072.stdout: osd scrub max interval = 600 2023-11-01T00:17:58.081 INFO:teuthology.orchestra.run.smithi072.stdout: osd mclock profile = high_recovery_ops 2023-11-01T00:17:58.081 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.082 INFO:teuthology.orchestra.run.smithi072.stdout: osd recover clone overlap = true 2023-11-01T00:17:58.082 INFO:teuthology.orchestra.run.smithi072.stdout: osd recovery max chunk = 1048576 2023-11-01T00:17:58.082 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.082 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug shutdown = true 2023-11-01T00:17:58.082 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug op order = true 2023-11-01T00:17:58.082 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug verify stray on activate = true 2023-11-01T00:17:58.083 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.083 INFO:teuthology.orchestra.run.smithi072.stdout: osd open classes on start = true 2023-11-01T00:17:58.083 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug pg log writeout = true 2023-11-01T00:17:58.083 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.083 INFO:teuthology.orchestra.run.smithi072.stdout: osd deep scrub update digest min age = 30 2023-11-01T00:17:58.084 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.084 INFO:teuthology.orchestra.run.smithi072.stdout: osd map max advance = 10 2023-11-01T00:17:58.084 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.084 INFO:teuthology.orchestra.run.smithi072.stdout: journal zero on create = true 2023-11-01T00:17:58.084 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.084 INFO:teuthology.orchestra.run.smithi072.stdout: filestore ondisk finisher threads = 3 2023-11-01T00:17:58.085 INFO:teuthology.orchestra.run.smithi072.stdout: filestore apply finisher threads = 3 2023-11-01T00:17:58.085 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.085 INFO:teuthology.orchestra.run.smithi072.stdout: bdev debug aio = true 2023-11-01T00:17:58.085 INFO:teuthology.orchestra.run.smithi072.stdout: osd debug misdirected ops = true 2023-11-01T00:17:58.085 INFO:teuthology.orchestra.run.smithi072.stdout: debug monc = 20 2023-11-01T00:17:58.085 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore block size = 96636764160 2023-11-01T00:17:58.086 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore compression algorithm = snappy 2023-11-01T00:17:58.086 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore compression mode = aggressive 2023-11-01T00:17:58.086 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore fsck on mount = True 2023-11-01T00:17:58.086 INFO:teuthology.orchestra.run.smithi072.stdout: bluestore rocksdb cf = False 2023-11-01T00:17:58.086 INFO:teuthology.orchestra.run.smithi072.stdout: crimson seastar smp = 3 2023-11-01T00:17:58.087 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluefs = 20 2023-11-01T00:17:58.087 INFO:teuthology.orchestra.run.smithi072.stdout: debug bluestore = 20 2023-11-01T00:17:58.087 INFO:teuthology.orchestra.run.smithi072.stdout: debug filestore = 0/0 2023-11-01T00:17:58.087 INFO:teuthology.orchestra.run.smithi072.stdout: debug journal = 0/0 2023-11-01T00:17:58.087 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 0/0 2023-11-01T00:17:58.087 INFO:teuthology.orchestra.run.smithi072.stdout: debug osd = 0/0 2023-11-01T00:17:58.088 INFO:teuthology.orchestra.run.smithi072.stdout: debug rocksdb = 10 2023-11-01T00:17:58.088 INFO:teuthology.orchestra.run.smithi072.stdout: err to stderr = True 2023-11-01T00:17:58.088 INFO:teuthology.orchestra.run.smithi072.stdout: log flush on exit = True 2023-11-01T00:17:58.088 INFO:teuthology.orchestra.run.smithi072.stdout: log to file = False 2023-11-01T00:17:58.088 INFO:teuthology.orchestra.run.smithi072.stdout: log to stderr = True 2023-11-01T00:17:58.088 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd backfillfull_ratio = 0.85 2023-11-01T00:17:58.089 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd full ratio = 0.9 2023-11-01T00:17:58.089 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd nearfull ratio = 0.8 2023-11-01T00:17:58.089 INFO:teuthology.orchestra.run.smithi072.stdout: osd failsafe full ratio = 0.95 2023-11-01T00:17:58.089 INFO:teuthology.orchestra.run.smithi072.stdout: osd objectstore = bluestore 2023-11-01T00:17:58.089 INFO:teuthology.orchestra.run.smithi072.stdout: osd shutdown pgref assert = True 2023-11-01T00:17:58.090 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.090 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr] 2023-11-01T00:17:58.090 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 1 2023-11-01T00:17:58.090 INFO:teuthology.orchestra.run.smithi072.stdout: debug mgr = 20 2023-11-01T00:17:58.090 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 20 2023-11-01T00:17:58.090 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2023-11-01T00:17:58.091 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2023-11-01T00:17:58.091 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2023-11-01T00:17:58.091 INFO:teuthology.orchestra.run.smithi072.stdout: mgr/telemetry/nag = false 2023-11-01T00:17:58.091 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.091 INFO:teuthology.orchestra.run.smithi072.stdout:[mon] 2023-11-01T00:17:58.091 INFO:teuthology.orchestra.run.smithi072.stdout: debug ms = 0/0 2023-11-01T00:17:58.092 INFO:teuthology.orchestra.run.smithi072.stdout: debug mon = 0/0 2023-11-01T00:17:58.092 INFO:teuthology.orchestra.run.smithi072.stdout: debug paxos = 0/0 2023-11-01T00:17:58.092 INFO:teuthology.orchestra.run.smithi072.stdout: debug auth = 20 2023-11-01T00:17:58.092 INFO:teuthology.orchestra.run.smithi072.stdout: mon data avail warn = 5 2023-11-01T00:17:58.092 INFO:teuthology.orchestra.run.smithi072.stdout: mon mgr mkfs grace = 240 2023-11-01T00:17:58.092 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min pgs per osd = 4 2023-11-01T00:17:58.093 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd reporter subtree level = osd 2023-11-01T00:17:58.093 INFO:teuthology.orchestra.run.smithi072.stdout: mon osd prime pg temp = true 2023-11-01T00:17:58.093 INFO:teuthology.orchestra.run.smithi072.stdout: mon reweight min bytes per osd = 10 2023-11-01T00:17:58.093 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.093 INFO:teuthology.orchestra.run.smithi072.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-11-01T00:17:58.093 INFO:teuthology.orchestra.run.smithi072.stdout: auth mon ticket ttl = 660 # 11m 2023-11-01T00:17:58.094 INFO:teuthology.orchestra.run.smithi072.stdout: auth service ticket ttl = 240 # 4m 2023-11-01T00:17:58.094 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.094 INFO:teuthology.orchestra.run.smithi072.stdout: # don't complain about insecure global_id in the test suite 2023-11-01T00:17:58.094 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-11-01T00:17:58.094 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-11-01T00:17:58.095 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.095 INFO:teuthology.orchestra.run.smithi072.stdout: # 1m isn't quite enough 2023-11-01T00:17:58.095 INFO:teuthology.orchestra.run.smithi072.stdout: mon_down_mkfs_grace = 2m 2023-11-01T00:17:58.095 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.095 INFO:teuthology.orchestra.run.smithi072.stdout: mon_warn_on_filestore_osds = false 2023-11-01T00:17:58.096 INFO:teuthology.orchestra.run.smithi072.stdout: osd pool default crimson = True 2023-11-01T00:17:58.097 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:17:58.097 INFO:teuthology.orchestra.run.smithi072.stdout:[client] 2023-11-01T00:17:58.097 INFO:teuthology.orchestra.run.smithi072.stdout: rgw cache enabled = true 2023-11-01T00:17:58.098 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable ops log = true 2023-11-01T00:17:58.098 INFO:teuthology.orchestra.run.smithi072.stdout: rgw enable usage log = true 2023-11-01T00:17:58.098 INFO:teuthology.orchestra.run.smithi072.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-11-01T00:17:58.098 INFO:teuthology.orchestra.run.smithi072.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-11-01T00:17:58.098 INFO:teuthology.orchestra.run.smithi072.stdout:[mon.a] 2023-11-01T00:17:58.099 INFO:tasks.ceph:Creating admin key on mon.a... 2023-11-01T00:17:58.100 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-11-01T00:17:58.200 INFO:tasks.ceph:Copying monmap to all nodes... 2023-11-01T00:17:58.201 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-11-01T00:17:58.201 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-11-01T00:17:58.215 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-11-01T00:17:58.216 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-11-01T00:17:58.270 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:17:58.270 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-11-01T00:17:58.270 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/ceph/ceph.keyring 2023-11-01T00:17:58.271 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-11-01T00:17:58.349 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-11-01T00:17:58.349 DEBUG:teuthology.orchestra.run.smithi072:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-11-01T00:17:58.403 INFO:tasks.ceph:Setting up mon nodes... 2023-11-01T00:17:58.404 INFO:tasks.ceph:Setting up mgr nodes... 2023-11-01T00:17:58.404 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-11-01T00:17:58.503 INFO:teuthology.orchestra.run.smithi072.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-11-01T00:17:58.506 INFO:tasks.ceph:Setting up mds nodes... 2023-11-01T00:17:58.507 INFO:tasks.ceph_client:Setting up client nodes... 2023-11-01T00:17:58.507 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-11-01T00:17:58.554 INFO:teuthology.orchestra.run.smithi072.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-11-01T00:17:58.570 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-11-01T00:17:58.570 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi072.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-11-01T00:17:58.570 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-11-01T00:17:58.597 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-01T00:17:58.597 INFO:tasks.ceph:role: osd.0 2023-11-01T00:17:58.597 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:17:58.597 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-11-01T00:17:58.670 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-11-01T00:17:58.671 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-01T00:17:58.671 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-01T00:17:58.671 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-01T00:17:58.671 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-01T00:17:58.671 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2023-11-01T00:17:58.672 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-01T00:17:58.672 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-01T00:17:58.672 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-01T00:17:58.672 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-01T00:17:59.179 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2023-11-01T00:17:59.181 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2023-11-01T00:17:59.181 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-11-01T00:17:59.293 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-11-01T00:17:59.328 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-11-01T00:17:59.395 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-01T00:17:59.395 INFO:tasks.ceph:role: osd.1 2023-11-01T00:17:59.395 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:17:59.396 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-11-01T00:17:59.469 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-11-01T00:17:59.469 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-01T00:17:59.469 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-01T00:17:59.470 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-01T00:17:59.470 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-01T00:17:59.470 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2023-11-01T00:17:59.470 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-01T00:17:59.470 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-01T00:17:59.471 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-01T00:17:59.471 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-01T00:17:59.930 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2023-11-01T00:17:59.932 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2023-11-01T00:17:59.933 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-11-01T00:18:00.020 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-11-01T00:18:00.051 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-11-01T00:18:00.118 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-11-01T00:18:00.118 INFO:tasks.ceph:role: osd.2 2023-11-01T00:18:00.119 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:18:00.119 DEBUG:teuthology.orchestra.run.smithi072:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-11-01T00:18:00.191 INFO:teuthology.orchestra.run.smithi072.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-11-01T00:18:00.192 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 attr=2, projid32bit=1 2023-11-01T00:18:00.192 INFO:teuthology.orchestra.run.smithi072.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-11-01T00:18:00.192 INFO:teuthology.orchestra.run.smithi072.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-11-01T00:18:00.192 INFO:teuthology.orchestra.run.smithi072.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-11-01T00:18:00.192 INFO:teuthology.orchestra.run.smithi072.stdout: = sunit=0 swidth=0 blks 2023-11-01T00:18:00.193 INFO:teuthology.orchestra.run.smithi072.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-11-01T00:18:00.193 INFO:teuthology.orchestra.run.smithi072.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-11-01T00:18:00.193 INFO:teuthology.orchestra.run.smithi072.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-11-01T00:18:00.193 INFO:teuthology.orchestra.run.smithi072.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-11-01T00:18:00.685 INFO:teuthology.orchestra.run.smithi072.stdout:Discarding blocks...Done. 2023-11-01T00:18:00.687 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi072.front.sepia.ceph.com -o noatime 2023-11-01T00:18:00.687 DEBUG:teuthology.orchestra.run.smithi072:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-11-01T00:18:00.820 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-11-01T00:18:00.851 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-01T00:18:01.360 INFO:teuthology.orchestra.run.smithi072.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-01T00:18:01.360 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:01,360 seastar - Reactor backend: linux-aio 2023-11-01T00:18:01.376 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:01,376 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-01T00:18:01.376 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:01,376 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-01T00:18:01.377 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:01,376 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-01T00:18:01.377 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:01,376 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-01T00:18:01.378 INFO:teuthology.orchestra.run.smithi072.stderr:==79803==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-01T00:18:01.385 INFO:teuthology.orchestra.run.smithi072.stderr:==79803==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe75bc7000; bottom 0x7f53886a8000; size: 0x00aaed51f000 (734126010368) 2023-11-01T00:18:01.385 INFO:teuthology.orchestra.run.smithi072.stderr:False positive error reports may follow 2023-11-01T00:18:01.385 INFO:teuthology.orchestra.run.smithi072.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-01T00:18:01.915 INFO:teuthology.orchestra.run.smithi072.stderr:----------------------------------------------------- 2023-11-01T00:18:01.915 INFO:teuthology.orchestra.run.smithi072.stderr:Suppressions used: 2023-11-01T00:18:01.915 INFO:teuthology.orchestra.run.smithi072.stderr: count bytes template 2023-11-01T00:18:01.915 INFO:teuthology.orchestra.run.smithi072.stderr: 1 8 MallocExtension::Register 2023-11-01T00:18:01.916 INFO:teuthology.orchestra.run.smithi072.stderr:----------------------------------------------------- 2023-11-01T00:18:01.916 INFO:teuthology.orchestra.run.smithi072.stderr: 2023-11-01T00:18:01.972 INFO:teuthology.orchestra.run.smithi072.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-01T00:18:01.972 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:01,972 seastar - Reactor backend: linux-aio 2023-11-01T00:18:02.008 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:02,007 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-01T00:18:02.008 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:02,007 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-01T00:18:02.008 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:02,007 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-01T00:18:02.008 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:02,007 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-01T00:18:02.018 INFO:teuthology.orchestra.run.smithi072.stderr:==79801==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-01T00:18:02.024 INFO:teuthology.orchestra.run.smithi072.stderr:==79801==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe75bc7000; bottom 0x7f5386492000; size: 0x00aaef735000 (734161752064) 2023-11-01T00:18:02.024 INFO:teuthology.orchestra.run.smithi072.stderr:False positive error reports may follow 2023-11-01T00:18:02.024 INFO:teuthology.orchestra.run.smithi072.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-01T00:18:02.107 INFO:teuthology.orchestra.run.smithi072.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c4bf7d6 0x2c4c0b84 0x2c4c20d2 0x2c4c2456 0x2c4c25a0 0x2c4c29f6 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x2604eb84 0x260511cd 0x26052fde 0x26055843 0x25e854b7 0x1eb9e762 0x1eb136d3 0x1eb1763f 0x2c84465b 2023-11-01T00:18:02.107 INFO:teuthology.orchestra.run.smithi072.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa2662817 0xffffffffa265e167 0xffffffffa28000aa 2023-11-01T00:18:06.041 INFO:teuthology.orchestra.run.smithi072.stderr:----------------------------------------------------- 2023-11-01T00:18:06.042 INFO:teuthology.orchestra.run.smithi072.stderr:Suppressions used: 2023-11-01T00:18:06.042 INFO:teuthology.orchestra.run.smithi072.stderr: count bytes template 2023-11-01T00:18:06.042 INFO:teuthology.orchestra.run.smithi072.stderr: 1 8 MallocExtension::Register 2023-11-01T00:18:06.043 INFO:teuthology.orchestra.run.smithi072.stderr:----------------------------------------------------- 2023-11-01T00:18:06.043 INFO:teuthology.orchestra.run.smithi072.stderr: 2023-11-01T00:18:06.043 INFO:teuthology.orchestra.run.smithi072.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid ef82487d-963f-4ecc-9718-b8d0ac62573e 2023-11-01T00:18:06.076 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-11-01T00:18:06.103 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-01T00:18:06.596 INFO:teuthology.orchestra.run.smithi072.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-01T00:18:06.596 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:06,596 seastar - Reactor backend: linux-aio 2023-11-01T00:18:06.606 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:06,606 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-01T00:18:06.607 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:06,606 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-01T00:18:06.607 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:06,606 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-01T00:18:06.607 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:06,606 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-01T00:18:06.608 INFO:teuthology.orchestra.run.smithi072.stderr:==79985==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-01T00:18:06.615 INFO:teuthology.orchestra.run.smithi072.stderr:==79985==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd807da000; bottom 0x7f07ded69000; size: 0x00f5a1a71000 (1054979067904) 2023-11-01T00:18:06.615 INFO:teuthology.orchestra.run.smithi072.stderr:False positive error reports may follow 2023-11-01T00:18:06.615 INFO:teuthology.orchestra.run.smithi072.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-01T00:18:06.641 INFO:teuthology.orchestra.run.smithi072.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c4bf7d6 0x2c4c0b84 0x2c4c20d2 0x2c4c2456 0x2c4c25a0 0x2c4c29f6 0x54daf 0x2c605 0x2857c 0xb6323 0x22891ac7 0x22890118 0x22890118 0x22890118 0x22a190d2 0x22a25105 0x22c4f1ed 0x22c5151f 0x229b6079 0x22a8ffff 0x22a92428 0x22a7eb74 0x22a7e1cc 0x22920b9e 0x247b7df3 0x247bd62a 0x247bfef4 0x247c10cd 0x2244bd9f 0x22485620 0x224c0899 0x2c461af1 0x2c4ff8fe 0x2c724930 0x2c72677b 0x2c17cdda 0x2c17ecdb 0x1ec5c758 0x1ec668c5 0x1eae33c9 0x3feaf 0x3ff5f 0x1e59ba04 2023-11-01T00:18:06.641 INFO:teuthology.orchestra.run.smithi072.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa1d93c59 0xffffffffa1d9acf9 0xffffffffa1d9b757 0xffffffffa1dc168f 0xffffffffa1d6ae85 0xffffffffa1d742a5 0xffffffffa1d74445 0xffffffffa1a89464 0xffffffffa2662562 0xffffffffa2800b82 2023-11-01T00:18:07.147 INFO:teuthology.orchestra.run.smithi072.stderr:----------------------------------------------------- 2023-11-01T00:18:07.147 INFO:teuthology.orchestra.run.smithi072.stderr:Suppressions used: 2023-11-01T00:18:07.147 INFO:teuthology.orchestra.run.smithi072.stderr: count bytes template 2023-11-01T00:18:07.147 INFO:teuthology.orchestra.run.smithi072.stderr: 1 8 MallocExtension::Register 2023-11-01T00:18:07.147 INFO:teuthology.orchestra.run.smithi072.stderr:----------------------------------------------------- 2023-11-01T00:18:07.148 INFO:teuthology.orchestra.run.smithi072.stderr: 2023-11-01T00:18:07.201 INFO:teuthology.orchestra.run.smithi072.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-01T00:18:07.201 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:07,201 seastar - Reactor backend: linux-aio 2023-11-01T00:18:07.219 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:07,218 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-01T00:18:07.219 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:07,218 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-01T00:18:07.219 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:07,219 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-01T00:18:07.226 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:07,226 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-01T00:18:07.229 INFO:teuthology.orchestra.run.smithi072.stderr:==79983==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-01T00:18:07.235 INFO:teuthology.orchestra.run.smithi072.stderr:==79983==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd807da000; bottom 0x7f07dcb52000; size: 0x00f5a3c88000 (1055014813696) 2023-11-01T00:18:07.235 INFO:teuthology.orchestra.run.smithi072.stderr:False positive error reports may follow 2023-11-01T00:18:07.235 INFO:teuthology.orchestra.run.smithi072.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-01T00:18:11.240 INFO:teuthology.orchestra.run.smithi072.stderr:----------------------------------------------------- 2023-11-01T00:18:11.241 INFO:teuthology.orchestra.run.smithi072.stderr:Suppressions used: 2023-11-01T00:18:11.241 INFO:teuthology.orchestra.run.smithi072.stderr: count bytes template 2023-11-01T00:18:11.241 INFO:teuthology.orchestra.run.smithi072.stderr: 1 8 MallocExtension::Register 2023-11-01T00:18:11.241 INFO:teuthology.orchestra.run.smithi072.stderr:----------------------------------------------------- 2023-11-01T00:18:11.242 INFO:teuthology.orchestra.run.smithi072.stderr: 2023-11-01T00:18:11.242 INFO:teuthology.orchestra.run.smithi072.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid ef82487d-963f-4ecc-9718-b8d0ac62573e 2023-11-01T00:18:11.290 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-11-01T00:18:11.317 DEBUG:teuthology.orchestra.run.smithi072:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-11-01T00:18:11.821 INFO:teuthology.orchestra.run.smithi072.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-01T00:18:11.822 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:11,821 seastar - Reactor backend: linux-aio 2023-11-01T00:18:11.837 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:11,837 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-01T00:18:11.838 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:11,837 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-01T00:18:11.838 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:11,837 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-01T00:18:11.838 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:11,837 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-01T00:18:11.839 INFO:teuthology.orchestra.run.smithi072.stderr:==80167==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-01T00:18:11.845 INFO:teuthology.orchestra.run.smithi072.stderr:==80167==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffefc379000; bottom 0x7fda5bca1000; size: 0x0024a06d8000 (157310353408) 2023-11-01T00:18:11.846 INFO:teuthology.orchestra.run.smithi072.stderr:False positive error reports may follow 2023-11-01T00:18:11.846 INFO:teuthology.orchestra.run.smithi072.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-01T00:18:11.872 INFO:teuthology.orchestra.run.smithi072.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2c4bf7d6 0x2c4c0b84 0x2c4c20d2 0x2c4c2456 0x2c4c25a0 0x2c4c29f6 0x54daf 0xb734c 0x22ae3086 0x2289ea02 0x22921e0f 0x247b7df3 0x247bd62a 0x247bfef4 0x247c10cd 0x2244bd9f 0x22485620 0x224c0899 0x2c461af1 0x2c4ff8fe 0x2c724930 0x2c72677b 0x2c17cdda 0x2c17ecdb 0x1ec5c758 0x1ec668c5 0x1eae33c9 0x3feaf 0x3ff5f 0x1e59ba04 2023-11-01T00:18:11.872 INFO:teuthology.orchestra.run.smithi072.stderr:kernel callstack: 2023-11-01T00:18:12.361 INFO:teuthology.orchestra.run.smithi072.stderr:----------------------------------------------------- 2023-11-01T00:18:12.361 INFO:teuthology.orchestra.run.smithi072.stderr:Suppressions used: 2023-11-01T00:18:12.361 INFO:teuthology.orchestra.run.smithi072.stderr: count bytes template 2023-11-01T00:18:12.362 INFO:teuthology.orchestra.run.smithi072.stderr: 1 8 MallocExtension::Register 2023-11-01T00:18:12.362 INFO:teuthology.orchestra.run.smithi072.stderr:----------------------------------------------------- 2023-11-01T00:18:12.362 INFO:teuthology.orchestra.run.smithi072.stderr: 2023-11-01T00:18:12.416 INFO:teuthology.orchestra.run.smithi072.stderr:WARNING: debug mode. Not for benchmarking or production 2023-11-01T00:18:12.417 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:12,417 seastar - Reactor backend: linux-aio 2023-11-01T00:18:12.435 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:12,434 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2023-11-01T00:18:12.435 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:12,434 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2023-11-01T00:18:12.435 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:12,434 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2023-11-01T00:18:12.442 INFO:teuthology.orchestra.run.smithi072.stderr:INFO 2023-11-01 00:18:12,442 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2023-11-01T00:18:12.445 INFO:teuthology.orchestra.run.smithi072.stderr:==80165==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2023-11-01T00:18:12.451 INFO:teuthology.orchestra.run.smithi072.stderr:==80165==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffefc379000; bottom 0x7fda59a8a000; size: 0x0024a28ef000 (157346099200) 2023-11-01T00:18:12.451 INFO:teuthology.orchestra.run.smithi072.stderr:False positive error reports may follow 2023-11-01T00:18:12.451 INFO:teuthology.orchestra.run.smithi072.stderr:For details see https://github.com/google/sanitizers/issues/189 2023-11-01T00:18:16.464 INFO:teuthology.orchestra.run.smithi072.stderr:----------------------------------------------------- 2023-11-01T00:18:16.464 INFO:teuthology.orchestra.run.smithi072.stderr:Suppressions used: 2023-11-01T00:18:16.464 INFO:teuthology.orchestra.run.smithi072.stderr: count bytes template 2023-11-01T00:18:16.464 INFO:teuthology.orchestra.run.smithi072.stderr: 1 8 MallocExtension::Register 2023-11-01T00:18:16.465 INFO:teuthology.orchestra.run.smithi072.stderr:----------------------------------------------------- 2023-11-01T00:18:16.465 INFO:teuthology.orchestra.run.smithi072.stderr: 2023-11-01T00:18:16.465 INFO:teuthology.orchestra.run.smithi072.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid ef82487d-963f-4ecc-9718-b8d0ac62573e 2023-11-01T00:18:16.500 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-11-01T00:18:16.527 INFO:tasks.ceph:Reading keys from all nodes... 2023-11-01T00:18:16.527 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-11-01T00:18:16.528 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-11-01T00:18:16.593 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-11-01T00:18:16.593 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-11-01T00:18:16.659 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-11-01T00:18:16.659 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-11-01T00:18:16.725 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-11-01T00:18:16.725 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-11-01T00:18:16.791 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-11-01T00:18:16.791 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-11-01T00:18:16.845 INFO:tasks.ceph:Adding keys to all mons... 2023-11-01T00:18:16.846 DEBUG:teuthology.orchestra.run.smithi072:> sudo tee -a /etc/ceph/ceph.keyring 2023-11-01T00:18:16.910 INFO:teuthology.orchestra.run.smithi072.stdout:[mgr.x] 2023-11-01T00:18:16.911 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQA2mUFlBD8HHhAAezW9ZSgzh3X4HKSBw4Fg2Q== 2023-11-01T00:18:16.911 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.0] 2023-11-01T00:18:16.911 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQA6mUFlpFIqBBAAKVY0j++40ZdSnKMGYx83jQ== 2023-11-01T00:18:16.911 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.1] 2023-11-01T00:18:16.912 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQA/mUFlilzCEBAA5GIdrOfU/9s1MyPbUjZ1Fw== 2023-11-01T00:18:16.912 INFO:teuthology.orchestra.run.smithi072.stdout:[osd.2] 2023-11-01T00:18:16.912 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQBEmUFl1E+SHRAAXn1+bk2iNvlz+mfByQJQVQ== 2023-11-01T00:18:16.912 INFO:teuthology.orchestra.run.smithi072.stdout:[client.0] 2023-11-01T00:18:16.913 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQA2mUFlfagPIRAAeFNgzjc2NkrRKzHrEx/2kg== 2023-11-01T00:18:16.914 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-11-01T00:18:17.011 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T00:18:17.068 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T00:18:17.125 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-11-01T00:18:17.183 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-11-01T00:18:17.242 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-11-01T00:18:17.243 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-11-01T00:18:17.269 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-11-01T00:18:17.370 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:17.368+0000 7f18d7831980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:17.371 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:17.368+0000 7f18d7831980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:17.386 DEBUG:teuthology.orchestra.run.smithi072:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-11-01T00:18:17.412 DEBUG:teuthology.orchestra.run.smithi072:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-11-01T00:18:17.467 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-11-01T00:18:17.467 INFO:tasks.ceph.mon.a:Restarting daemon 2023-11-01T00:18:17.468 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-11-01T00:18:17.509 INFO:tasks.ceph.mon.a:Started 2023-11-01T00:18:17.509 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-11-01T00:18:17.510 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-11-01T00:18:17.510 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-11-01T00:18:17.512 INFO:tasks.ceph.mgr.x:Started 2023-11-01T00:18:17.513 INFO:tasks.ceph:Setting crush tunables to default 2023-11-01T00:18:17.513 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph osd crush tunables default 2023-11-01T00:18:17.589 INFO:tasks.ceph.mon.a.smithi072.stderr:2023-11-01T00:18:17.587+0000 7fcb0ae0d980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:17.590 INFO:tasks.ceph.mon.a.smithi072.stderr:2023-11-01T00:18:17.587+0000 7fcb0ae0d980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:17.590 INFO:tasks.ceph.mon.a.smithi072.stderr:2023-11-01T00:18:17.587+0000 7fcb0ae0d980 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:17.591 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:17.588+0000 7f6a79566f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:17.620 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:17.613+0000 7f44f6b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:17.657 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:17.655+0000 7f44f6b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:17.724 INFO:teuthology.orchestra.run.smithi072.stderr:adjusted tunables profile to default 2023-11-01T00:18:17.735 INFO:tasks.ceph:check_enable_crimson: True 2023-11-01T00:18:17.735 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2023-11-01T00:18:17.736 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2023-11-01T00:18:17.793 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:17.791+0000 7f6a79566f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:17.794 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:17.791+0000 7f6a79566f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:17.840 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:17.834+0000 7fc731b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:17.841 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:17.839+0000 7fc731b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:17.920 INFO:tasks.ceph.mon.a.smithi072.stderr:2023-11-01T00:18:17.918+0000 7fcb02df1640 -1 WARNING: experimental feature 'crimson' is enabled 2023-11-01T00:18:17.921 INFO:tasks.ceph.mon.a.smithi072.stderr:Please be aware that this feature is experimental, untested, 2023-11-01T00:18:17.921 INFO:tasks.ceph.mon.a.smithi072.stderr:unsupported, and may result in data corruption, data loss, 2023-11-01T00:18:17.921 INFO:tasks.ceph.mon.a.smithi072.stderr:and/or irreparable damage to your cluster. Do not use 2023-11-01T00:18:17.921 INFO:tasks.ceph.mon.a.smithi072.stderr:feature with important data. 2023-11-01T00:18:17.922 INFO:tasks.ceph.mon.a.smithi072.stderr: 2023-11-01T00:18:18.266 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:18.264+0000 7f6a79566f00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-01T00:18:18.316 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:18.314+0000 7f6a79566f00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-01T00:18:18.737 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-11-01T00:18:18.738 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-11-01T00:18:18.738 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-11-01T00:18:18.765 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-11-01T00:18:18.765 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-11-01T00:18:18.832 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-11-01T00:18:18.832 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-11-01T00:18:18.852 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:18.851+0000 7f6a79566f00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-01T00:18:18.878 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph osd new 2a9f66aa-b317-4b77-ac15-a4c241d33a90 0 2023-11-01T00:18:18.907 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:18.906+0000 7f6a79566f00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-01T00:18:18.954 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:18.953+0000 7f6a79566f00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-01T00:18:19.011 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:19.010+0000 7f6a79566f00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-01T00:18:19.012 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:19.006+0000 7fab13ad5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:19.014 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:19.013+0000 7fab13ad5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:19.055 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:19.054+0000 7f6a79566f00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-01T00:18:19.090 INFO:teuthology.orchestra.run.smithi072.stdout:0 2023-11-01T00:18:19.100 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph osd new 30cdc914-2c8e-493d-80df-e935e5bb1a6f 1 2023-11-01T00:18:19.160 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:19.159+0000 7f6a79566f00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-01T00:18:19.205 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:19.199+0000 7f5bd951e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:19.206 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:19.206+0000 7f5bd951e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:19.234 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:19.233+0000 7f6a79566f00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-01T00:18:19.279 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:19.278+0000 7f6a79566f00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-01T00:18:19.280 INFO:teuthology.orchestra.run.smithi072.stdout:1 2023-11-01T00:18:19.291 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph osd new 4da9f42c-89df-4666-ae2c-1564c18f103b 2 2023-11-01T00:18:19.334 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:19.333+0000 7f6a79566f00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-01T00:18:19.384 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:19.383+0000 7f6a79566f00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-01T00:18:19.396 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:19.390+0000 7f891ae24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:19.398 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:19.397+0000 7f891ae24640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:19.474 INFO:teuthology.orchestra.run.smithi072.stdout:2 2023-11-01T00:18:19.485 DEBUG:teuthology.orchestra.run.smithi072:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2023-11-01T00:18:19.556 INFO:tasks.ceph.osd.0:Restarting daemon 2023-11-01T00:18:19.557 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2023-11-01T00:18:19.599 INFO:tasks.ceph.osd.0:Started 2023-11-01T00:18:19.599 DEBUG:teuthology.orchestra.run.smithi072:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2023-11-01T00:18:19.630 INFO:tasks.ceph.osd.1:Restarting daemon 2023-11-01T00:18:19.631 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2023-11-01T00:18:19.673 INFO:tasks.ceph.osd.1:Started 2023-11-01T00:18:19.673 DEBUG:teuthology.orchestra.run.smithi072:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2023-11-01T00:18:19.706 INFO:tasks.ceph.osd.2:Restarting daemon 2023-11-01T00:18:19.706 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2023-11-01T00:18:19.748 INFO:tasks.ceph.osd.2:Started 2023-11-01T00:18:19.749 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-01T00:18:19.801 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:19.800+0000 7f6a79566f00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-01T00:18:19.949 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:19.941+0000 7f0dbc06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:19.953 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:19.952+0000 7f0dbc06d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:19.967 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:19.966+0000 7f6a79566f00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-01T00:18:20.082 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:18:20.083 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":6,"fsid":"ef82487d-963f-4ecc-9718-b8d0ac62573e","created":"2023-11-01T00:18:17.648984+0000","modified":"2023-11-01T00:18:19.473141+0000","last_up_change":"0.000000","last_in_change":"2023-11-01T00:18:19.473141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"2a9f66aa-b317-4b77-ac15-a4c241d33a90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"30cdc914-2c8e-493d-80df-e935e5bb1a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4da9f42c-89df-4666-ae2c-1564c18f103b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-01T00:18:20.097 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:20.096+0000 7f6a79566f00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-01T00:18:20.104 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-11-01T00:18:20.105 INFO:tasks.ceph:Waiting for OSDs to come up 2023-11-01T00:18:20.152 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:20.151+0000 7f6a79566f00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-01T00:18:20.258 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:20.257+0000 7f6a79566f00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-01T00:18:20.357 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:20.355+0000 7f6a79566f00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-01T00:18:20.407 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-01T00:18:20.504 INFO:teuthology.misc.health.smithi072.stderr:2023-11-01T00:18:20.500+0000 7f03bc6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:20.505 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:20.504+0000 7f6a79566f00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-01T00:18:20.506 INFO:teuthology.misc.health.smithi072.stderr:2023-11-01T00:18:20.505+0000 7f03bc6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:20.552 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:20.551+0000 7f6a79566f00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-01T00:18:20.572 INFO:teuthology.misc.health.smithi072.stdout: 2023-11-01T00:18:20.572 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":6,"fsid":"ef82487d-963f-4ecc-9718-b8d0ac62573e","created":"2023-11-01T00:18:17.648984+0000","modified":"2023-11-01T00:18:19.473141+0000","last_up_change":"0.000000","last_in_change":"2023-11-01T00:18:19.473141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"2a9f66aa-b317-4b77-ac15-a4c241d33a90","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"30cdc914-2c8e-493d-80df-e935e5bb1a6f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4da9f42c-89df-4666-ae2c-1564c18f103b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-01T00:18:20.583 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-11-01T00:18:20.707 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:20.706+0000 7f6a79566f00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-01T00:18:21.606 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:21.605+0000 7f6a79566f00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-01T00:18:21.904 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:21.902+0000 7f6a79566f00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-01T00:18:21.960 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:21.959+0000 7f6a79566f00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-01T00:18:22.066 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:22.065+0000 7f6a79566f00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-01T00:18:23.085 INFO:tasks.ceph.mgr.x.smithi072.stderr:2023-11-01T00:18:23.084+0000 7f6a191d1640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-11-01T00:18:26.885 DEBUG:teuthology.orchestra.run.smithi072:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-11-01T00:18:26.993 INFO:teuthology.misc.health.smithi072.stderr:2023-11-01T00:18:26.988+0000 7fd58462d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:26.995 INFO:teuthology.misc.health.smithi072.stderr:2023-11-01T00:18:26.993+0000 7fd58462d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:27.190 INFO:teuthology.misc.health.smithi072.stdout: 2023-11-01T00:18:27.190 INFO:teuthology.misc.health.smithi072.stdout:{"epoch":9,"fsid":"ef82487d-963f-4ecc-9718-b8d0ac62573e","created":"2023-11-01T00:18:17.648984+0000","modified":"2023-11-01T00:18:27.092073+0000","last_up_change":"2023-11-01T00:18:25.086232+0000","last_in_change":"2023-11-01T00:18:19.473141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-01T00:18:26.123226+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2a9f66aa-b317-4b77-ac15-a4c241d33a90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6803","nonce":1365074895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1365074895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6811","nonce":1365074895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":1365074895}]},"public_addr":"172.21.15.72:6803/1365074895","cluster_addr":"172.21.15.72:6802/1365074895","heartbeat_back_addr":"172.21.15.72:6811/1365074895","heartbeat_front_addr":"172.21.15.72:6808/1365074895","state":["exists","up"]},{"osd":1,"uuid":"30cdc914-2c8e-493d-80df-e935e5bb1a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2149100712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6805","nonce":2149100712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6809","nonce":2149100712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":2149100712}]},"public_addr":"172.21.15.72:6804/2149100712","cluster_addr":"172.21.15.72:6805/2149100712","heartbeat_back_addr":"172.21.15.72:6809/2149100712","heartbeat_front_addr":"172.21.15.72:6810/2149100712","state":["exists","up"]},{"osd":2,"uuid":"4da9f42c-89df-4666-ae2c-1564c18f103b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2296573133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6807","nonce":2296573133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":2296573133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6813","nonce":2296573133}]},"public_addr":"172.21.15.72:6806/2296573133","cluster_addr":"172.21.15.72:6807/2296573133","heartbeat_back_addr":"172.21.15.72:6812/2296573133","heartbeat_front_addr":"172.21.15.72:6813/2296573133","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-01T00:18:27.200 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-11-01T00:18:27.201 INFO:tasks.ceph:Creating RBD pool 2023-11-01T00:18:27.201 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph osd pool create rbd 8 2023-11-01T00:18:27.305 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:27.298+0000 7f0af411e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:27.307 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:27.306+0000 7f0af411e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:28.099 INFO:teuthology.orchestra.run.smithi072.stderr:pool 'rbd' created 2023-11-01T00:18:28.114 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-11-01T00:18:28.234 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:28.224+0000 7f96fcd1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:28.236 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:28.235+0000 7f96fcd1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:29.105 INFO:teuthology.orchestra.run.smithi072.stderr:enabled application 'rbd' on pool 'rbd' 2023-11-01T00:18:29.119 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-11-01T00:18:29.120 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-11-01T00:18:29.120 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-11-01T00:18:29.120 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-11-01T00:18:29.121 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-11-01T00:18:29.249 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:29.245+0000 7fa81bd5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:29.250 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:29.249+0000 7fa81bd5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:29.473 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:18:29.483 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":4,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":3693252377},{"type":"v1","addr":"172.21.15.72:6801","nonce":3693252377}]},"active_addr":"172.21.15.72:6801/3693252377","active_change":"2023-11-01T00:18:22.078478+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":2336070922}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":2150584666}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":1281599153}]}]} 2023-11-01T00:18:29.486 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-11-01T00:18:29.486 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-11-01T00:18:29.486 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-01T00:18:29.591 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:29.587+0000 7fe122f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:29.593 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:29.592+0000 7fe122f9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:29.767 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:18:29.768 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":11,"fsid":"ef82487d-963f-4ecc-9718-b8d0ac62573e","created":"2023-11-01T00:18:17.648984+0000","modified":"2023-11-01T00:18:29.101995+0000","last_up_change":"2023-11-01T00:18:25.086232+0000","last_in_change":"2023-11-01T00:18:19.473141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-01T00:18:26.123226+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-01T00:18:27.496872+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2a9f66aa-b317-4b77-ac15-a4c241d33a90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6803","nonce":1365074895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1365074895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6811","nonce":1365074895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":1365074895}]},"public_addr":"172.21.15.72:6803/1365074895","cluster_addr":"172.21.15.72:6802/1365074895","heartbeat_back_addr":"172.21.15.72:6811/1365074895","heartbeat_front_addr":"172.21.15.72:6808/1365074895","state":["exists","up"]},{"osd":1,"uuid":"30cdc914-2c8e-493d-80df-e935e5bb1a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2149100712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6805","nonce":2149100712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6809","nonce":2149100712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":2149100712}]},"public_addr":"172.21.15.72:6804/2149100712","cluster_addr":"172.21.15.72:6805/2149100712","heartbeat_back_addr":"172.21.15.72:6809/2149100712","heartbeat_front_addr":"172.21.15.72:6810/2149100712","state":["exists","up"]},{"osd":2,"uuid":"4da9f42c-89df-4666-ae2c-1564c18f103b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2296573133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6807","nonce":2296573133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":2296573133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6813","nonce":2296573133}]},"public_addr":"172.21.15.72:6806/2296573133","cluster_addr":"172.21.15.72:6807/2296573133","heartbeat_back_addr":"172.21.15.72:6812/2296573133","heartbeat_front_addr":"172.21.15.72:6813/2296573133","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-01T00:18:29.779 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-11-01T00:18:29.780 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-11-01T00:18:29.884 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:29.880+0000 7ffb6646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:29.886 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:29.885+0000 7ffb6646d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:30.064 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:18:30.064 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":11,"fsid":"ef82487d-963f-4ecc-9718-b8d0ac62573e","created":"2023-11-01T00:18:17.648984+0000","modified":"2023-11-01T00:18:29.101995+0000","last_up_change":"2023-11-01T00:18:25.086232+0000","last_in_change":"2023-11-01T00:18:19.473141+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-01T00:18:26.123226+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-11-01T00:18:27.496872+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2a9f66aa-b317-4b77-ac15-a4c241d33a90","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6803","nonce":1365074895}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1365074895}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6811","nonce":1365074895}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":1365074895}]},"public_addr":"172.21.15.72:6803/1365074895","cluster_addr":"172.21.15.72:6802/1365074895","heartbeat_back_addr":"172.21.15.72:6811/1365074895","heartbeat_front_addr":"172.21.15.72:6808/1365074895","state":["exists","up"]},{"osd":1,"uuid":"30cdc914-2c8e-493d-80df-e935e5bb1a6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":2149100712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6805","nonce":2149100712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6809","nonce":2149100712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":2149100712}]},"public_addr":"172.21.15.72:6804/2149100712","cluster_addr":"172.21.15.72:6805/2149100712","heartbeat_back_addr":"172.21.15.72:6809/2149100712","heartbeat_front_addr":"172.21.15.72:6810/2149100712","state":["exists","up"]},{"osd":2,"uuid":"4da9f42c-89df-4666-ae2c-1564c18f103b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2296573133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6807","nonce":2296573133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":2296573133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6813","nonce":2296573133}]},"public_addr":"172.21.15.72:6806/2296573133","cluster_addr":"172.21.15.72:6807/2296573133","heartbeat_back_addr":"172.21.15.72:6812/2296573133","heartbeat_front_addr":"172.21.15.72:6813/2296573133","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-01T00:18:30.076 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-11-01T00:18:30.077 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-11-01T00:18:30.078 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-11-01T00:18:30.197 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:30.191+0000 7fce9959e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:30.198 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:30.192+0000 7f29646f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:30.198 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:30.192+0000 7fe07ff9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:30.199 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:30.198+0000 7fce9959e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:30.200 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:30.199+0000 7f29646f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:30.200 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:30.199+0000 7fe07ff9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:30.225 INFO:teuthology.orchestra.run.smithi072.stdout:34359738373 2023-11-01T00:18:30.226 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-11-01T00:18:30.227 INFO:teuthology.orchestra.run.smithi072.stdout:34359738373 2023-11-01T00:18:30.228 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-11-01T00:18:30.229 INFO:teuthology.orchestra.run.smithi072.stdout:34359738373 2023-11-01T00:18:30.229 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-01T00:18:30.335 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:30.330+0000 7fa3de264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:30.337 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:30.336+0000 7fa3de264640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:30.378 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:30.373+0000 7f424e115640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:30.380 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:30.379+0000 7f424e115640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:30.392 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:30.387+0000 7ff5fa99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:30.394 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:30.393+0000 7ff5fa99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:30.555 INFO:teuthology.orchestra.run.smithi072.stdout:34359738373 2023-11-01T00:18:30.565 INFO:teuthology.orchestra.run.smithi072.stdout:34359738372 2023-11-01T00:18:30.567 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2023-11-01T00:18:30.567 DEBUG:teuthology.parallel:result is None 2023-11-01T00:18:30.578 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.2 2023-11-01T00:18:30.598 INFO:teuthology.orchestra.run.smithi072.stdout:34359738373 2023-11-01T00:18:30.610 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2023-11-01T00:18:30.610 DEBUG:teuthology.parallel:result is None 2023-11-01T00:18:31.579 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-01T00:18:31.685 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:31.680+0000 7f1ecb99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:31.686 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:31.686+0000 7f1ecb99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:31.858 INFO:teuthology.orchestra.run.smithi072.stdout:34359738372 2023-11-01T00:18:31.871 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.2 2023-11-01T00:18:32.872 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-11-01T00:18:32.991 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:32.985+0000 7f2b2d6d5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:32.992 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:32.991+0000 7f2b2d6d5640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:33.168 INFO:teuthology.orchestra.run.smithi072.stdout:34359738373 2023-11-01T00:18:33.180 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2023-11-01T00:18:33.181 DEBUG:teuthology.parallel:result is None 2023-11-01T00:18:33.181 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-11-01T00:18:33.181 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-01T00:18:33.285 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:33.281+0000 7f0e8b99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:33.287 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:33.286+0000 7f0e8b99e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:33.461 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:18:33.461 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2023-11-01T00:18:33.472 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-11-01T00:18:32.082598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80764,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034756,"statfs":{"total":289910292480,"available":289827590144,"internally_reserved":0,"allocated":274432,"data_stored":1227066,"data_compressed":59932,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.869877"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-01T00:18:30.144186+0000","last_change":"2023-11-01T00:18:29.185306+0000","last_active":"2023-11-01T00:18:30.144186+0000","last_peered":"2023-11-01T00:18:30.144186+0000","last_clean":"2023-11-01T00:18:30.144186+0000","last_became_active":"2023-11-01T00:18:29.181698+0000","last_became_peered":"2023-11-01T00:18:29.181698+0000","last_unstale":"2023-11-01T00:18:30.144186+0000","last_undegraded":"2023-11-01T00:18:30.144186+0000","last_fullsized":"2023-11-01T00:18:30.144186+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_clean_scrub_stamp":"2023-11-01T00:18:28.097557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-01T00:18:30.143637+0000","last_change":"2023-11-01T00:18:29.171191+0000","last_active":"2023-11-01T00:18:30.143637+0000","last_peered":"2023-11-01T00:18:30.143637+0000","last_clean":"2023-11-01T00:18:30.143637+0000","last_became_active":"2023-11-01T00:18:29.165574+0000","last_became_peered":"2023-11-01T00:18:29.165574+0000","last_unstale":"2023-11-01T00:18:30.143637+0000","last_undegraded":"2023-11-01T00:18:30.143637+0000","last_fullsized":"2023-11-01T00:18:30.143637+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_clean_scrub_stamp":"2023-11-01T00:18:28.097557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-01T00:18:30.142520+0000","last_change":"2023-11-01T00:18:29.207331+0000","last_active":"2023-11-01T00:18:30.142520+0000","last_peered":"2023-11-01T00:18:30.142520+0000","last_clean":"2023-11-01T00:18:30.142520+0000","last_became_active":"2023-11-01T00:18:29.204985+0000","last_became_peered":"2023-11-01T00:18:29.204985+0000","last_unstale":"2023-11-01T00:18:30.142520+0000","last_undegraded":"2023-11-01T00:18:30.142520+0000","last_fullsized":"2023-11-01T00:18:30.142520+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_clean_scrub_stamp":"2023-11-01T00:18:28.097557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-01T00:18:30.144726+0000","last_change":"2023-11-01T00:18:29.221411+0000","last_active":"2023-11-01T00:18:30.144726+0000","last_peered":"2023-11-01T00:18:30.144726+0000","last_clean":"2023-11-01T00:18:30.144726+0000","last_became_active":"2023-11-01T00:18:29.210795+0000","last_became_peered":"2023-11-01T00:18:29.210795+0000","last_unstale":"2023-11-01T00:18:30.144726+0000","last_undegraded":"2023-11-01T00:18:30.144726+0000","last_fullsized":"2023-11-01T00:18:30.144726+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_clean_scrub_stamp":"2023-11-01T00:18:28.097557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-01T00:18:30.145979+0000","last_change":"2023-11-01T00:18:29.193848+0000","last_active":"2023-11-01T00:18:30.145979+0000","last_peered":"2023-11-01T00:18:30.145979+0000","last_clean":"2023-11-01T00:18:30.145979+0000","last_became_active":"2023-11-01T00:18:29.191648+0000","last_became_peered":"2023-11-01T00:18:29.191648+0000","last_unstale":"2023-11-01T00:18:30.145979+0000","last_undegraded":"2023-11-01T00:18:30.145979+0000","last_fullsized":"2023-11-01T00:18:30.145979+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_clean_scrub_stamp":"2023-11-01T00:18:28.097557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-01T00:18:30.114389+0000","last_change":"2023-11-01T00:18:29.197588+0000","last_active":"2023-11-01T00:18:30.114389+0000","last_peered":"2023-11-01T00:18:30.114389+0000","last_clean":"2023-11-01T00:18:30.114389+0000","last_became_active":"2023-11-01T00:18:29.195302+0000","last_became_peered":"2023-11-01T00:18:29.195302+0000","last_unstale":"2023-11-01T00:18:30.114389+0000","last_undegraded":"2023-11-01T00:18:30.114389+0000","last_fullsized":"2023-11-01T00:18:30.114389+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_clean_scrub_stamp":"2023-11-01T00:18:28.097557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-01T00:18:30.114116+0000","last_change":"2023-11-01T00:18:29.177346+0000","last_active":"2023-11-01T00:18:30.114116+0000","last_peered":"2023-11-01T00:18:30.114116+0000","last_clean":"2023-11-01T00:18:30.114116+0000","last_became_active":"2023-11-01T00:18:29.175163+0000","last_became_peered":"2023-11-01T00:18:29.175163+0000","last_unstale":"2023-11-01T00:18:30.114116+0000","last_undegraded":"2023-11-01T00:18:30.114116+0000","last_fullsized":"2023-11-01T00:18:30.114116+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_clean_scrub_stamp":"2023-11-01T00:18:28.097557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-01T00:18:30.148217+0000","last_change":"2023-11-01T00:18:29.186554+0000","last_active":"2023-11-01T00:18:30.148217+0000","last_peered":"2023-11-01T00:18:30.148217+0000","last_clean":"2023-11-01T00:18:30.148217+0000","last_became_active":"2023-11-01T00:18:29.182382+0000","last_became_peered":"2023-11-01T00:18:29.182382+0000","last_unstale":"2023-11-01T00:18:30.148217+0000","last_undegraded":"2023-11-01T00:18:30.148217+0000","last_fullsized":"2023-11-01T00:18:30.148217+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_clean_scrub_stamp":"2023-11-01T00:18:28.097557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-01T00:18:30.140154+0000","last_change":"2023-11-01T00:18:28.162088+0000","last_active":"2023-11-01T00:18:30.140154+0000","last_peered":"2023-11-01T00:18:30.140154+0000","last_clean":"2023-11-01T00:18:30.140154+0000","last_became_active":"2023-11-01T00:18:28.154621+0000","last_became_peered":"2023-11-01T00:18:28.154621+0000","last_unstale":"2023-11-01T00:18:30.140154+0000","last_undegraded":"2023-11-01T00:18:30.140154+0000","last_fullsized":"2023-11-01T00:18:30.140154+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T00:18:27.092073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T00:18:27.092073+0000","last_clean_scrub_stamp":"2023-11-01T00:18:27.092073+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":16984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29966,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29966,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-01T00:18:33.473 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-11-01T00:18:33.578 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:33.574+0000 7f109259e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:33.580 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:33.579+0000 7f109259e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:33.752 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:18:33.753 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2023-11-01T00:18:33.764 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2023-11-01T00:18:32.082598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80764,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034756,"statfs":{"total":289910292480,"available":289827590144,"internally_reserved":0,"allocated":274432,"data_stored":1227066,"data_compressed":59932,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.869877"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-01T00:18:30.144186+0000","last_change":"2023-11-01T00:18:29.185306+0000","last_active":"2023-11-01T00:18:30.144186+0000","last_peered":"2023-11-01T00:18:30.144186+0000","last_clean":"2023-11-01T00:18:30.144186+0000","last_became_active":"2023-11-01T00:18:29.181698+0000","last_became_peered":"2023-11-01T00:18:29.181698+0000","last_unstale":"2023-11-01T00:18:30.144186+0000","last_undegraded":"2023-11-01T00:18:30.144186+0000","last_fullsized":"2023-11-01T00:18:30.144186+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_clean_scrub_stamp":"2023-11-01T00:18:28.097557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-01T00:18:30.143637+0000","last_change":"2023-11-01T00:18:29.171191+0000","last_active":"2023-11-01T00:18:30.143637+0000","last_peered":"2023-11-01T00:18:30.143637+0000","last_clean":"2023-11-01T00:18:30.143637+0000","last_became_active":"2023-11-01T00:18:29.165574+0000","last_became_peered":"2023-11-01T00:18:29.165574+0000","last_unstale":"2023-11-01T00:18:30.143637+0000","last_undegraded":"2023-11-01T00:18:30.143637+0000","last_fullsized":"2023-11-01T00:18:30.143637+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_clean_scrub_stamp":"2023-11-01T00:18:28.097557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-01T00:18:30.142520+0000","last_change":"2023-11-01T00:18:29.207331+0000","last_active":"2023-11-01T00:18:30.142520+0000","last_peered":"2023-11-01T00:18:30.142520+0000","last_clean":"2023-11-01T00:18:30.142520+0000","last_became_active":"2023-11-01T00:18:29.204985+0000","last_became_peered":"2023-11-01T00:18:29.204985+0000","last_unstale":"2023-11-01T00:18:30.142520+0000","last_undegraded":"2023-11-01T00:18:30.142520+0000","last_fullsized":"2023-11-01T00:18:30.142520+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_clean_scrub_stamp":"2023-11-01T00:18:28.097557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-01T00:18:30.144726+0000","last_change":"2023-11-01T00:18:29.221411+0000","last_active":"2023-11-01T00:18:30.144726+0000","last_peered":"2023-11-01T00:18:30.144726+0000","last_clean":"2023-11-01T00:18:30.144726+0000","last_became_active":"2023-11-01T00:18:29.210795+0000","last_became_peered":"2023-11-01T00:18:29.210795+0000","last_unstale":"2023-11-01T00:18:30.144726+0000","last_undegraded":"2023-11-01T00:18:30.144726+0000","last_fullsized":"2023-11-01T00:18:30.144726+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_clean_scrub_stamp":"2023-11-01T00:18:28.097557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-01T00:18:30.145979+0000","last_change":"2023-11-01T00:18:29.193848+0000","last_active":"2023-11-01T00:18:30.145979+0000","last_peered":"2023-11-01T00:18:30.145979+0000","last_clean":"2023-11-01T00:18:30.145979+0000","last_became_active":"2023-11-01T00:18:29.191648+0000","last_became_peered":"2023-11-01T00:18:29.191648+0000","last_unstale":"2023-11-01T00:18:30.145979+0000","last_undegraded":"2023-11-01T00:18:30.145979+0000","last_fullsized":"2023-11-01T00:18:30.145979+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_clean_scrub_stamp":"2023-11-01T00:18:28.097557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-01T00:18:30.114389+0000","last_change":"2023-11-01T00:18:29.197588+0000","last_active":"2023-11-01T00:18:30.114389+0000","last_peered":"2023-11-01T00:18:30.114389+0000","last_clean":"2023-11-01T00:18:30.114389+0000","last_became_active":"2023-11-01T00:18:29.195302+0000","last_became_peered":"2023-11-01T00:18:29.195302+0000","last_unstale":"2023-11-01T00:18:30.114389+0000","last_undegraded":"2023-11-01T00:18:30.114389+0000","last_fullsized":"2023-11-01T00:18:30.114389+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_clean_scrub_stamp":"2023-11-01T00:18:28.097557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-01T00:18:30.114116+0000","last_change":"2023-11-01T00:18:29.177346+0000","last_active":"2023-11-01T00:18:30.114116+0000","last_peered":"2023-11-01T00:18:30.114116+0000","last_clean":"2023-11-01T00:18:30.114116+0000","last_became_active":"2023-11-01T00:18:29.175163+0000","last_became_peered":"2023-11-01T00:18:29.175163+0000","last_unstale":"2023-11-01T00:18:30.114116+0000","last_undegraded":"2023-11-01T00:18:30.114116+0000","last_fullsized":"2023-11-01T00:18:30.114116+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_clean_scrub_stamp":"2023-11-01T00:18:28.097557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-01T00:18:30.148217+0000","last_change":"2023-11-01T00:18:29.186554+0000","last_active":"2023-11-01T00:18:30.148217+0000","last_peered":"2023-11-01T00:18:30.148217+0000","last_clean":"2023-11-01T00:18:30.148217+0000","last_became_active":"2023-11-01T00:18:29.182382+0000","last_became_peered":"2023-11-01T00:18:29.182382+0000","last_unstale":"2023-11-01T00:18:30.148217+0000","last_undegraded":"2023-11-01T00:18:30.148217+0000","last_fullsized":"2023-11-01T00:18:30.148217+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T00:18:28.097557+0000","last_clean_scrub_stamp":"2023-11-01T00:18:28.097557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-11-01T00:18:30.140154+0000","last_change":"2023-11-01T00:18:28.162088+0000","last_active":"2023-11-01T00:18:30.140154+0000","last_peered":"2023-11-01T00:18:30.140154+0000","last_clean":"2023-11-01T00:18:30.140154+0000","last_became_active":"2023-11-01T00:18:28.154621+0000","last_became_peered":"2023-11-01T00:18:28.154621+0000","last_unstale":"2023-11-01T00:18:30.140154+0000","last_undegraded":"2023-11-01T00:18:30.140154+0000","last_fullsized":"2023-11-01T00:18:30.140154+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-11-01T00:18:27.092073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-11-01T00:18:27.092073+0000","last_clean_scrub_stamp":"2023-11-01T00:18:27.092073+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26876,"kb_used_data":52,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344964,"statfs":{"total":96636764160,"available":96609243136,"internally_reserved":0,"allocated":53248,"data_stored":16984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29966,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26944,"kb_used_data":108,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344896,"statfs":{"total":96636764160,"available":96609173504,"internally_reserved":0,"allocated":110592,"data_stored":605041,"data_compressed":29966,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2023-11-01T00:18:33.765 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-11-01T00:18:33.765 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-11-01T00:18:33.765 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-11-01T00:18:33.766 DEBUG:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-11-01T00:18:33.869 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:33.866+0000 7fe4ca4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:33.871 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:18:33.870+0000 7fe4ca4ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:18:34.080 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:18:34.080 INFO:teuthology.orchestra.run.smithi072.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-11-01T00:18:34.092 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-11-01T00:18:34.093 INFO:teuthology.run_tasks:Running task cbt... 2023-11-01T00:18:34.125 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi072.front.sepia.ceph.com', 'osds': ['ubuntu@smithi072.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi072.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi072.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi072.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'replicated': {'pg_size': 256, 'pgp_size': 256, 'replication': 'replicated'}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'radosbench': {'concurrent_ops': 4, 'concurrent_procs': 2, 'op_size': [4194304], 'pool_monitoring_list': ['collectl'], 'pool_profile': 'replicated', 'run_monitoring_list': ['collectl'], 'time': 60, 'write_only': True}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2023-11-01T00:18:34.126 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2023-11-01T00:18:34.144 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-11-01T00:18:34.145 DEBUG:teuthology.orchestra.run.smithi072:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-11-01T00:18:34.197 DEBUG:teuthology.orchestra.run.smithi072:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2023-11-01T00:18:34.250 INFO:teuthology.orchestra.run.smithi072.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2023-11-01T00:18:34.645 INFO:tasks.cbt:Installing collectl 2023-11-01T00:18:34.646 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2023-11-01T00:18:34.773 INFO:teuthology.orchestra.run.smithi072.stderr:--2023-11-01 00:18:34-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2023-11-01T00:18:34.980 INFO:teuthology.orchestra.run.smithi072.stderr:Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::ac40:9691, ... 2023-11-01T00:18:34.990 INFO:teuthology.orchestra.run.smithi072.stderr:Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. 2023-11-01T00:18:35.257 INFO:teuthology.orchestra.run.smithi072.stderr:HTTP request sent, awaiting response... 302 Found 2023-11-01T00:18:35.257 INFO:teuthology.orchestra.run.smithi072.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlQZlbbtyrHCbsmfT4kL60w9CLLRjCNWtjymUvr5YWUSm-s2N6oo41G7UsEwmcZO_JomFStsbCK93EM8STuTHSWqisaw%3D%3D&use_mirror=phoenixnap&r= [following] 2023-11-01T00:18:35.257 INFO:teuthology.orchestra.run.smithi072.stderr:--2023-11-01 00:18:35-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlQZlbbtyrHCbsmfT4kL60w9CLLRjCNWtjymUvr5YWUSm-s2N6oo41G7UsEwmcZO_JomFStsbCK93EM8STuTHSWqisaw%3D%3D&use_mirror=phoenixnap&r= 2023-11-01T00:18:35.258 INFO:teuthology.orchestra.run.smithi072.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2023-11-01T00:18:35.328 INFO:teuthology.orchestra.run.smithi072.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2023-11-01T00:18:35.562 INFO:teuthology.orchestra.run.smithi072.stderr:HTTP request sent, awaiting response... 302 Found 2023-11-01T00:18:35.562 INFO:teuthology.orchestra.run.smithi072.stderr:Location: https://phoenixnap.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2023-11-01T00:18:35.562 INFO:teuthology.orchestra.run.smithi072.stderr:--2023-11-01 00:18:35-- https://phoenixnap.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2023-11-01T00:18:35.563 INFO:teuthology.orchestra.run.smithi072.stderr:Resolving phoenixnap.dl.sourceforge.net (phoenixnap.dl.sourceforge.net)... 184.164.141.26 2023-11-01T00:18:35.630 INFO:teuthology.orchestra.run.smithi072.stderr:Connecting to phoenixnap.dl.sourceforge.net (phoenixnap.dl.sourceforge.net)|184.164.141.26|:443... connected. 2023-11-01T00:18:35.939 INFO:teuthology.orchestra.run.smithi072.stderr:HTTP request sent, awaiting response... 200 OK 2023-11-01T00:18:35.940 INFO:teuthology.orchestra.run.smithi072.stderr:Length: 620693 (606K) [application/x-gzip] 2023-11-01T00:18:35.940 INFO:teuthology.orchestra.run.smithi072.stderr:Saving to: ‘collectl.tar.gz’ 2023-11-01T00:18:35.940 INFO:teuthology.orchestra.run.smithi072.stderr: 2023-11-01T00:18:36.006 INFO:teuthology.orchestra.run.smithi072.stderr: 0K .......... .......... .......... .......... .......... 8% 748K 1s 2023-11-01T00:18:36.073 INFO:teuthology.orchestra.run.smithi072.stderr: 50K .......... .......... .......... .......... .......... 16% 751K 1s 2023-11-01T00:18:36.073 INFO:teuthology.orchestra.run.smithi072.stderr: 100K .......... .......... .......... .......... .......... 24% 114M 0s 2023-11-01T00:18:36.141 INFO:teuthology.orchestra.run.smithi072.stderr: 150K .......... .......... .......... .......... .......... 32% 755K 0s 2023-11-01T00:18:36.141 INFO:teuthology.orchestra.run.smithi072.stderr: 200K .......... .......... .......... .......... .......... 41% 136M 0s 2023-11-01T00:18:36.141 INFO:teuthology.orchestra.run.smithi072.stderr: 250K .......... .......... .......... .......... .......... 49% 117M 0s 2023-11-01T00:18:36.142 INFO:teuthology.orchestra.run.smithi072.stderr: 300K .......... .......... .......... .......... .......... 57% 114M 0s 2023-11-01T00:18:36.208 INFO:teuthology.orchestra.run.smithi072.stderr: 350K .......... .......... .......... .......... .......... 65% 761K 0s 2023-11-01T00:18:36.208 INFO:teuthology.orchestra.run.smithi072.stderr: 400K .......... .......... .......... .......... .......... 74% 146M 0s 2023-11-01T00:18:36.209 INFO:teuthology.orchestra.run.smithi072.stderr: 450K .......... .......... .......... .......... .......... 82% 145M 0s 2023-11-01T00:18:36.209 INFO:teuthology.orchestra.run.smithi072.stderr: 500K .......... .......... .......... .......... .......... 90% 115M 0s 2023-11-01T00:18:36.209 INFO:teuthology.orchestra.run.smithi072.stderr: 550K .......... .......... .......... .......... .......... 98% 113M 0s 2023-11-01T00:18:36.209 INFO:teuthology.orchestra.run.smithi072.stderr: 600K ...... 100% 11.4T=0.3s 2023-11-01T00:18:36.209 INFO:teuthology.orchestra.run.smithi072.stderr: 2023-11-01T00:18:36.210 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01 00:18:36 (2.20 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2023-11-01T00:18:36.210 INFO:teuthology.orchestra.run.smithi072.stderr: 2023-11-01T00:18:36.245 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/ 2023-11-01T00:18:36.246 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/collectl.conf 2023-11-01T00:18:36.246 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/ 2023-11-01T00:18:36.246 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/SlowProc.html 2023-11-01T00:18:36.246 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/BuddyInfo.html 2023-11-01T00:18:36.247 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/WhySummary.html 2023-11-01T00:18:36.247 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Socket.html 2023-11-01T00:18:36.247 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/colmux.html 2023-11-01T00:18:36.247 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Tutorial.html 2023-11-01T00:18:36.247 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Gexpr.html 2023-11-01T00:18:36.248 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Import.html 2023-11-01T00:18:36.248 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/style.css 2023-11-01T00:18:36.248 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2023-11-01T00:18:36.248 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2023-11-01T00:18:36.248 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/FileNaming.html 2023-11-01T00:18:36.248 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/TheMath.html 2023-11-01T00:18:36.249 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Environmental.html 2023-11-01T00:18:36.249 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Process.html 2023-11-01T00:18:36.249 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/OutputFiles.html 2023-11-01T00:18:36.249 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/HiResTime.html 2023-11-01T00:18:36.249 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/OutputFormats.html 2023-11-01T00:18:36.250 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/slub.jpg 2023-11-01T00:18:36.250 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Lustre.html 2023-11-01T00:18:36.250 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Documentation.html 2023-11-01T00:18:36.250 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Inodes.html 2023-11-01T00:18:36.250 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Graphite.html 2023-11-01T00:18:36.251 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/InputFiles.html 2023-11-01T00:18:36.251 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Export.html 2023-11-01T00:18:36.251 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Interrupts.html 2023-11-01T00:18:36.251 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Startup.html 2023-11-01T00:18:36.251 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Performance.html 2023-11-01T00:18:36.251 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Messages.html 2023-11-01T00:18:36.252 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Logging.html 2023-11-01T00:18:36.252 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Data-detail.html 2023-11-01T00:18:36.252 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/CPUs.html 2023-11-01T00:18:36.252 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Matrix.html 2023-11-01T00:18:36.252 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Data-brief.html 2023-11-01T00:18:36.253 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/SlabInfo.html 2023-11-01T00:18:36.253 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Misc.html 2023-11-01T00:18:36.253 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Examples.html 2023-11-01T00:18:36.253 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/OpenStack.html 2023-11-01T00:18:36.253 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Plotfiles.html 2023-11-01T00:18:36.254 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2023-11-01T00:18:36.254 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2023-11-01T00:18:36.254 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Data-verbose.html 2023-11-01T00:18:36.254 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Data.html 2023-11-01T00:18:36.254 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Exceptions.html 2023-11-01T00:18:36.254 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/RunningAsAService.html 2023-11-01T00:18:36.255 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2023-11-01T00:18:36.255 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Ganglia.jpg 2023-11-01T00:18:36.255 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/OperationalModes.html 2023-11-01T00:18:36.255 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Playback.html 2023-11-01T00:18:36.256 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Hello.html 2023-11-01T00:18:36.256 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Sockets.html 2023-11-01T00:18:36.256 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Network.html 2023-11-01T00:18:36.256 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Infiniband.html 2023-11-01T00:18:36.256 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Memory.html 2023-11-01T00:18:36.257 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2023-11-01T00:18:36.257 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/Disks.html 2023-11-01T00:18:36.257 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/docs/NfsInfo.html 2023-11-01T00:18:36.257 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/service/ 2023-11-01T00:18:36.257 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/service/collectl.service 2023-11-01T00:18:36.258 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/initd/ 2023-11-01T00:18:36.258 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/initd/collectl-suse 2023-11-01T00:18:36.258 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/initd/collectl-debian 2023-11-01T00:18:36.258 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/initd/collectl-generic 2023-11-01T00:18:36.258 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/initd/collectl 2023-11-01T00:18:36.259 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/misc.ph 2023-11-01T00:18:36.259 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/gexpr.ph 2023-11-01T00:18:36.259 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/statsd.ph 2023-11-01T00:18:36.259 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/envrules.std 2023-11-01T00:18:36.259 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/hello.ph 2023-11-01T00:18:36.259 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/vmsum.ph 2023-11-01T00:18:36.260 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/proctree.ph 2023-11-01T00:18:36.260 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/README-WINDOWS 2023-11-01T00:18:36.260 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/formatit.ph 2023-11-01T00:18:36.262 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/colmux 2023-11-01T00:18:36.262 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/INSTALL 2023-11-01T00:18:36.262 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/graphite.ph 2023-11-01T00:18:36.262 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/RELEASE-collectl 2023-11-01T00:18:36.263 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/vmstat.ph 2023-11-01T00:18:36.263 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/README 2023-11-01T00:18:36.263 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/man1/ 2023-11-01T00:18:36.263 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/man1/collectl.1 2023-11-01T00:18:36.264 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/man1/colmux.1 2023-11-01T00:18:36.264 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/lexpr.ph 2023-11-01T00:18:36.264 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/client.pl 2023-11-01T00:18:36.264 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/collectl 2023-11-01T00:18:36.264 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/COPYING 2023-11-01T00:18:36.265 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/vnet.ph 2023-11-01T00:18:36.265 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/GPL 2023-11-01T00:18:36.265 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/UNINSTALL 2023-11-01T00:18:36.265 INFO:teuthology.orchestra.run.smithi072.stdout:collectl-4.3.1/ARTISTIC 2023-11-01T00:18:36.303 INFO:teuthology.orchestra.run.smithi072.stderr:./INSTALL: line 90: chkconfig: command not found 2023-11-01T00:18:36.304 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2023-11-01T00:18:36.846 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:03:47 ago on Wed 01 Nov 2023 12:14:49 AM UTC. 2023-11-01T00:18:36.940 INFO:teuthology.orchestra.run.smithi072.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2023-11-01T00:18:36.942 INFO:teuthology.orchestra.run.smithi072.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2023-11-01T00:18:36.996 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2023-11-01T00:18:36.997 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:18:36.997 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2023-11-01T00:18:36.998 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:18:36.998 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2023-11-01T00:18:36.998 INFO:teuthology.orchestra.run.smithi072.stdout: librbd-devel x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 23 k 2023-11-01T00:18:36.998 INFO:teuthology.orchestra.run.smithi072.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2023-11-01T00:18:36.998 INFO:teuthology.orchestra.run.smithi072.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2023-11-01T00:18:36.999 INFO:teuthology.orchestra.run.smithi072.stdout: perf x86_64 5.14.0-378.el9 appstream 7.3 M 2023-11-01T00:18:36.999 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2023-11-01T00:18:36.999 INFO:teuthology.orchestra.run.smithi072.stdout: libradospp-devel x86_64 2:18.0.0-6795.gd1dcabba.el9 ceph 30 k 2023-11-01T00:18:36.999 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:18:36.999 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2023-11-01T00:18:36.999 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2023-11-01T00:18:37.000 INFO:teuthology.orchestra.run.smithi072.stdout:Install 5 Packages 2023-11-01T00:18:37.000 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:18:37.000 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 7.5 M 2023-11-01T00:18:37.001 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 11 M 2023-11-01T00:18:37.001 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2023-11-01T00:18:37.361 INFO:teuthology.orchestra.run.smithi072.stdout:(1/5): librbd-devel-18.0.0-6795.gd1dcabba.el9.x 128 kB/s | 23 kB 00:00 2023-11-01T00:18:37.430 INFO:teuthology.orchestra.run.smithi072.stdout:(2/5): libradospp-devel-18.0.0-6795.gd1dcabba.e 121 kB/s | 30 kB 00:00 2023-11-01T00:18:37.880 INFO:teuthology.orchestra.run.smithi072.stdout:(3/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 29 kB/s | 13 kB 00:00 2023-11-01T00:18:38.022 INFO:teuthology.orchestra.run.smithi072.stdout:(4/5): perf-5.14.0-378.el9.x86_64.rpm 8.7 MB/s | 7.3 MB 00:00 2023-11-01T00:18:38.064 INFO:teuthology.orchestra.run.smithi072.stdout:(5/5): pdsh-2.34-7.el9.x86_64.rpm 169 kB/s | 118 kB 00:00 2023-11-01T00:18:38.067 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2023-11-01T00:18:38.067 INFO:teuthology.orchestra.run.smithi072.stdout:Total 7.0 MB/s | 7.5 MB 00:01 2023-11-01T00:18:38.144 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2023-11-01T00:18:38.165 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2023-11-01T00:18:38.165 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2023-11-01T00:18:38.364 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2023-11-01T00:18:38.365 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2023-11-01T00:18:38.709 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2023-11-01T00:18:38.783 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2023-11-01T00:18:38.831 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2023-11-01T00:18:38.863 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libradospp-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 3/5 2023-11-01T00:18:39.228 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librbd-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 4/5 2023-11-01T00:18:39.389 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : perf-5.14.0-378.el9.x86_64 5/5 2023-11-01T00:18:40.533 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: perf-5.14.0-378.el9.x86_64 5/5 2023-11-01T00:18:40.534 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libradospp-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 1/5 2023-11-01T00:18:40.534 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librbd-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 2/5 2023-11-01T00:18:40.534 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : perf-5.14.0-378.el9.x86_64 3/5 2023-11-01T00:18:40.534 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2023-11-01T00:18:40.884 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2023-11-01T00:18:40.884 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:18:40.885 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2023-11-01T00:18:40.885 INFO:teuthology.orchestra.run.smithi072.stdout: libradospp-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:18:40.885 INFO:teuthology.orchestra.run.smithi072.stdout: librbd-devel-2:18.0.0-6795.gd1dcabba.el9.x86_64 2023-11-01T00:18:40.885 INFO:teuthology.orchestra.run.smithi072.stdout: pdsh-2.34-7.el9.x86_64 2023-11-01T00:18:40.885 INFO:teuthology.orchestra.run.smithi072.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2023-11-01T00:18:40.886 INFO:teuthology.orchestra.run.smithi072.stdout: perf-5.14.0-378.el9.x86_64 2023-11-01T00:18:40.886 INFO:teuthology.orchestra.run.smithi072.stdout: 2023-11-01T00:18:40.886 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2023-11-01T00:18:41.413 INFO:tasks.cbt:benchmark: radosbench 2023-11-01T00:18:41.413 DEBUG:teuthology.orchestra.run.smithi072:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2023-11-01T00:18:41.442 DEBUG:teuthology.orchestra.run.smithi072:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2023-11-01T00:18:41.855 INFO:teuthology.orchestra.run.smithi072.stderr:00:18:41 - DEBUG - cbt - Settings.cluster: 2023-11-01T00:18:41.855 INFO:teuthology.orchestra.run.smithi072.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2023-11-01T00:18:41.855 INFO:teuthology.orchestra.run.smithi072.stderr: 'clients': ['ubuntu@smithi072.front.sepia.ceph.com'], 2023-11-01T00:18:41.855 INFO:teuthology.orchestra.run.smithi072.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2023-11-01T00:18:41.856 INFO:teuthology.orchestra.run.smithi072.stderr: 'head': 'ubuntu@smithi072.front.sepia.ceph.com', 2023-11-01T00:18:41.856 INFO:teuthology.orchestra.run.smithi072.stderr: 'is_teuthology': True, 2023-11-01T00:18:41.856 INFO:teuthology.orchestra.run.smithi072.stderr: 'iterations': 1, 2023-11-01T00:18:41.856 INFO:teuthology.orchestra.run.smithi072.stderr: 'mons': ['ubuntu@smithi072.front.sepia.ceph.com'], 2023-11-01T00:18:41.856 INFO:teuthology.orchestra.run.smithi072.stderr: 'osds': ['ubuntu@smithi072.front.sepia.ceph.com'], 2023-11-01T00:18:41.856 INFO:teuthology.orchestra.run.smithi072.stderr: 'osds_per_node': 3, 2023-11-01T00:18:41.857 INFO:teuthology.orchestra.run.smithi072.stderr: 'pid_dir': '/var/run/ceph', 2023-11-01T00:18:41.857 INFO:teuthology.orchestra.run.smithi072.stderr: 'pool_profiles': {'replicated': {'pg_size': 256, 2023-11-01T00:18:41.857 INFO:teuthology.orchestra.run.smithi072.stderr: 'pgp_size': 256, 2023-11-01T00:18:41.857 INFO:teuthology.orchestra.run.smithi072.stderr: 'replication': 'replicated'}}, 2023-11-01T00:18:41.857 INFO:teuthology.orchestra.run.smithi072.stderr: 'rebuild_every_test': False, 2023-11-01T00:18:41.858 INFO:teuthology.orchestra.run.smithi072.stderr: 'rgws': {'ubuntu@smithi072.front.sepia.ceph.com': None}, 2023-11-01T00:18:41.858 INFO:teuthology.orchestra.run.smithi072.stderr: 'tmp_dir': '/tmp/cbt', 2023-11-01T00:18:41.858 INFO:teuthology.orchestra.run.smithi072.stderr: 'use_existing': True, 2023-11-01T00:18:41.858 INFO:teuthology.orchestra.run.smithi072.stderr: 'user': 'ubuntu'} 2023-11-01T00:18:41.874 INFO:teuthology.orchestra.run.smithi072.stderr:00:18:41 - INFO - cbt - Pausing for 60s for idle monitoring. 2023-11-01T00:18:41.876 INFO:teuthology.orchestra.run.smithi072.stderr:00:18:41 - DEBUG - cbt - Nodes : ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:18:41.876 INFO:teuthology.orchestra.run.smithi072.stderr:00:18:41 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:18:41.876 INFO:teuthology.orchestra.run.smithi072.stderr:00:18:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl 2023-11-01T00:18:41.879 INFO:teuthology.orchestra.run.smithi072.stderr:00:18:41 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:18:41.879 INFO:teuthology.orchestra.run.smithi072.stderr:00:18:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-11-01T00:18:41.880 INFO:teuthology.orchestra.run.smithi072.stderr:00:18:41 - DEBUG - cbt - Nodes : ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:18:41.880 INFO:teuthology.orchestra.run.smithi072.stderr:00:18:41 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:18:41.880 INFO:teuthology.orchestra.run.smithi072.stderr:00:18:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf 2023-11-01T00:18:41.883 INFO:teuthology.orchestra.run.smithi072.stderr:00:18:41 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:18:41.883 INFO:teuthology.orchestra.run.smithi072.stderr:00:18:41 - DEBUG - cbt - PerfMonitoring: in local_node 2023-11-01T00:18:41.883 INFO:teuthology.orchestra.run.smithi072.stderr:00:18:41 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-11-01T00:19:41.921 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:41 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:41.922 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-11-01T00:19:41.954 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:41 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:41.954 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-11-01T00:19:41.990 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:41 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:41.990 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf.data 2023-11-01T00:19:41.992 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:41 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:41.992 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:41 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/idle_monitoring/perf/perf_stat.* 2023-11-01T00:19:41.993 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:41 - DEBUG - cbt - Nodes : ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:19:41.994 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:41 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:41.994 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:41 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* 2023-11-01T00:19:42.010 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:42.010 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/$(basename ${i}).smithi072.front.sepia.ceph.com ; done 2023-11-01T00:19:42.014 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - Nodes : ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:19:42.015 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:42.015 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo killall -9 rados 2023-11-01T00:19:42.038 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - WARNING - cbt - sudo killall -9 rados 2023-11-01T00:19:42.038 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - WARNING - cbt - error 1 seen, continuing anyway... 2023-11-01T00:19:42.057 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3. 2023-11-01T00:19:42.057 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2023-11-01T00:19:42.057 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - Nodes : ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:19:42.057 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:42.058 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004 2023-11-01T00:19:42.073 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - Nodes : ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:19:42.073 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:42.073 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl 2023-11-01T00:19:42.076 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:42.076 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-11-01T00:19:42.077 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - Nodes : ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:19:42.077 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:42.078 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf 2023-11-01T00:19:42.080 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:42.080 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - PerfMonitoring: in local_node 2023-11-01T00:19:42.080 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-11-01T00:19:42.081 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - Nodes : ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:19:42.081 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:42.081 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete rados-bench-cbt rados-bench-cbt --yes-i-really-really-mean-it 2023-11-01T00:19:42.375 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - Nodes : ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:19:42.375 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:42.375 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:42 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create rados-bench-cbt 256 256 2023-11-01T00:19:43.632 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:43 - DEBUG - cbt - Nodes : ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:19:43.632 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:43 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:43.633 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:43 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable rados-bench-cbt radosbench 2023-11-01T00:19:44.634 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:44 - INFO - cbt - Checking Health after pool creation. 2023-11-01T00:19:49.640 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:49 - DEBUG - cbt - Nodes : ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:19:49.640 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:49 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:49.640 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:49 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-11-01T00:19:49.953 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:49 - INFO - cbt - Final Pool Health Check. 2023-11-01T00:19:54.959 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:54 - DEBUG - cbt - Nodes : ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:19:54.959 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:54 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:54.959 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:54 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2023-11-01T00:19:55.265 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:55 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:55.265 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-11-01T00:19:55.283 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:55 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:55.283 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-11-01T00:19:55.313 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:55 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:55.313 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf.data 2023-11-01T00:19:55.315 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:55 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:55.315 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/pool_monitoring/perf/perf_stat.* 2023-11-01T00:19:55.316 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:55 - DEBUG - cbt - Nodes : ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:19:55.316 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:55 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:55.317 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2023-11-01T00:19:55.436 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:55 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:55.436 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:55 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2023-11-01T00:19:56.039 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:56 - DEBUG - cbt - Nodes : ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:19:56.040 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:56 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:56.040 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -v 2023-11-01T00:19:56.445 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:56 - DEBUG - cbt - Nodes : ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:19:56.446 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:56 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:56.446 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:56 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write 2023-11-01T00:19:56.517 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:56 - DEBUG - cbt - Nodes : ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:19:56.517 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:56 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:56.518 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/ceph_settings.out 2023-11-01T00:19:57.565 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:57 - DEBUG - cbt - Nodes : ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:19:57.565 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:57 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:57.565 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl 2023-11-01T00:19:57.567 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:57 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:57.568 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2023-11-01T00:19:57.569 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:57 - DEBUG - cbt - Nodes : ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:19:57.569 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:57 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:57.570 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf 2023-11-01T00:19:57.572 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:57 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:57.572 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:57 - DEBUG - cbt - PerfMonitoring: in local_node 2023-11-01T00:19:57.573 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:57 - DEBUG - cbt - pid_dir: /var/run/ceph 2023-11-01T00:19:57.573 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:57 - INFO - cbt - Running radosbench write test. 2023-11-01T00:19:57.573 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:57 - DEBUG - cbt - Nodes : ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:19:57.573 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:57 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:57.573 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-0 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.0.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.0 2023-11-01T00:19:57.574 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:57 - DEBUG - cbt - Nodes : ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:19:57.574 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:57 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:19:57.575 INFO:teuthology.orchestra.run.smithi072.stderr:00:19:57 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/rados -c /etc/ceph/ceph.conf -p rados-bench-cbt bench -b 4194304 60 write --concurrent-ios 4 --run-name `hostname -f`-1 --no-cleanup 2> /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/objecter.1.log > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/output.1 2023-11-01T00:20:58.720 INFO:teuthology.orchestra.run.smithi072.stderr:00:20:58 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:20:58.721 INFO:teuthology.orchestra.run.smithi072.stderr:00:20:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2023-11-01T00:20:58.749 INFO:teuthology.orchestra.run.smithi072.stderr:00:20:58 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:20:58.750 INFO:teuthology.orchestra.run.smithi072.stderr:00:20:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2023-11-01T00:20:58.784 INFO:teuthology.orchestra.run.smithi072.stderr:00:20:58 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:20:58.784 INFO:teuthology.orchestra.run.smithi072.stderr:00:20:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf.data 2023-11-01T00:20:58.785 INFO:teuthology.orchestra.run.smithi072.stderr:00:20:58 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:20:58.786 INFO:teuthology.orchestra.run.smithi072.stderr:00:20:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/perf/perf_stat.* 2023-11-01T00:20:58.787 INFO:teuthology.orchestra.run.smithi072.stderr:00:20:58 - DEBUG - cbt - Nodes : ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:20:58.787 INFO:teuthology.orchestra.run.smithi072.stderr:00:20:58 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:20:58.787 INFO:teuthology.orchestra.run.smithi072.stderr:00:20:58 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/historic_ops.out 2023-11-01T00:20:59.080 INFO:teuthology.orchestra.run.smithi072.stderr:00:20:59 - DEBUG - cbt - Nodes : ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:20:59.106 INFO:teuthology.orchestra.run.smithi072.stderr:00:20:59 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:20:59.106 INFO:teuthology.orchestra.run.smithi072.stderr:00:20:59 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* 2023-11-01T00:20:59.123 INFO:teuthology.orchestra.run.smithi072.stderr:00:20:59 - DEBUG - cbt - get_fqdn_local()=smithi072.front.sepia.ceph.com 2023-11-01T00:20:59.123 INFO:teuthology.orchestra.run.smithi072.stderr:00:20:59 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/Radosbench/osd_ra-00000128/op_size-04194304/concurrent_ops-00000004/write/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/$(basename ${i}).smithi072.front.sepia.ceph.com ; done 2023-11-01T00:20:59.153 INFO:teuthology.orchestra.run.smithi072.stderr:00:20:59 - INFO - cbt - Convert results to json format. 2023-11-01T00:20:59.153 INFO:teuthology.orchestra.run.smithi072.stderr:00:20:59 - DEBUG - cbt - Nodes : ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:20:59.170 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2023-11-01T00:20:59.179 DEBUG:teuthology.orchestra.run.smithi072:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2023-11-01T00:20:59.387 INFO:tasks.cbt_performance:Collecting CBT performance data config 2023-11-01T00:20:59.388 INFO:tasks.cbt_performance:reading cbt preferences 2023-11-01T00:20:59.388 DEBUG:teuthology.orchestra.run.smithi072:> find /home/ubuntu/cephtest -name 'json_output.*' 2023-11-01T00:20:59.453 INFO:teuthology.orchestra.run.smithi072.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.0.smithi072.front.sepia.ceph.com 2023-11-01T00:20:59.453 INFO:teuthology.orchestra.run.smithi072.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.1.smithi072.front.sepia.ceph.com 2023-11-01T00:20:59.454 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-11-01T00:20:59.454 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.0.smithi072.front.sepia.ceph.com of=/dev/stdout 2023-11-01T00:20:59.477 INFO:tasks.cbt_performance:reading total cpu cycles 2023-11-01T00:20:59.478 DEBUG:teuthology.orchestra.run.smithi072:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write -name 'perf_stat.*' 2023-11-01T00:20:59.532 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-11-01T00:20:59.533 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-11-01T00:20:59.533 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2023-11-01T00:20:59.533 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write/json_output.1.smithi072.front.sepia.ceph.com of=/dev/stdout 2023-11-01T00:20:59.587 INFO:tasks.cbt_performance:reading total cpu cycles 2023-11-01T00:20:59.588 DEBUG:teuthology.orchestra.run.smithi072:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-9a441ab3/write -name 'perf_stat.*' 2023-11-01T00:20:59.642 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2023-11-01T00:20:59.642 INFO:tasks.cbt_performance:total cpu cycles: 0 2023-11-01T00:20:59.651 INFO:tasks.cbt_performance:Data inserted successfully. 2023-11-01T00:20:59.658 INFO:tasks.cbt_performance:Data inserted successfully. 2023-11-01T00:20:59.659 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-11-01T00:20:59.667 DEBUG:teuthology.orchestra.run.smithi072:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-11-01T00:20:59.795 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:20:59.788+0000 7f87d76ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:20:59.797 INFO:teuthology.orchestra.run.smithi072.stderr:2023-11-01T00:20:59.796+0000 7f87d76ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2023-11-01T00:20:59.989 INFO:teuthology.misc:Shutting down mds daemons... 2023-11-01T00:20:59.990 INFO:teuthology.misc:Shutting down osd daemons... 2023-11-01T00:20:59.990 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-11-01T00:20:59.991 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T00:21:00.084 INFO:tasks.ceph.osd.0:Stopped 2023-11-01T00:21:00.084 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-11-01T00:21:00.084 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T00:21:03.510 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-11-01T00:21:06.189 INFO:tasks.ceph.osd.1:Stopped 2023-11-01T00:21:06.190 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-11-01T00:21:06.190 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T00:21:08.712 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-11-01T00:21:08.713 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-11-01T00:21:12.293 INFO:tasks.ceph.osd.2:Stopped 2023-11-01T00:21:12.293 INFO:teuthology.misc:Shutting down mgr daemons... 2023-11-01T00:21:12.294 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-11-01T00:21:12.294 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T00:21:12.324 INFO:tasks.ceph.mgr.x:Stopped 2023-11-01T00:21:12.324 INFO:teuthology.misc:Shutting down mon daemons... 2023-11-01T00:21:12.325 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-11-01T00:21:12.325 INFO:teuthology.orchestra.run:waiting for 300 2023-11-01T00:21:12.376 INFO:tasks.ceph.mon.a:Stopped 2023-11-01T00:21:12.377 INFO:tasks.ceph:Checking cluster log for badness... 2023-11-01T00:21:12.377 DEBUG:teuthology.orchestra.run.smithi072:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-11-01T00:21:12.458 INFO:teuthology.orchestra.run.smithi072.stdout:2023-11-01T00:20:32.740993+0000 mon.a (mon.0) 155 : cluster [WRN] Health check failed: Degraded data redundancy: 11/1808 objects degraded (0.608%), 2 pgs degraded (PG_DEGRADED) 2023-11-01T00:21:12.459 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-11-01T00:21:12.459 DEBUG:teuthology.orchestra.run.smithi072:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-11-01T00:21:12.529 DEBUG:teuthology.orchestra.run.smithi072:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-11-01T00:21:12.598 DEBUG:teuthology.orchestra.run.smithi072:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-11-01T00:21:12.668 INFO:teuthology.orchestra.run.smithi072.stdout:2023-11-01T00:20:32.740993+0000 mon.a (mon.0) 155 : cluster [WRN] Health check failed: Degraded data redundancy: 11/1808 objects degraded (0.608%), 2 pgs degraded (PG_DEGRADED) 2023-11-01T00:21:12.668 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:21:12.668 DEBUG:teuthology.orchestra.run.smithi072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-11-01T00:21:13.495 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:21:13.496 DEBUG:teuthology.orchestra.run.smithi072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-11-01T00:21:13.628 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:21:13.628 DEBUG:teuthology.orchestra.run.smithi072:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-11-01T00:21:13.679 INFO:tasks.ceph:Archiving mon data... 2023-11-01T00:21:13.680 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi/7442572/data/mon.a.tgz 2023-11-01T00:21:13.681 DEBUG:teuthology.orchestra.run.smithi072:> mktemp 2023-11-01T00:21:13.704 INFO:teuthology.orchestra.run.smithi072.stdout:/tmp/tmp.R5Oay0rgLP 2023-11-01T00:21:13.704 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.R5Oay0rgLP 2023-11-01T00:21:13.714 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-11-01T00:21:13.864 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0666 /tmp/tmp.R5Oay0rgLP 2023-11-01T00:21:13.974 DEBUG:teuthology.orchestra.remote:smithi072:/tmp/tmp.R5Oay0rgLP is 180KB 2023-11-01T00:21:14.025 DEBUG:teuthology.orchestra.run.smithi072:> rm -fr /tmp/tmp.R5Oay0rgLP 2023-11-01T00:21:14.039 INFO:tasks.ceph:Cleaning ceph cluster... 2023-11-01T00:21:14.039 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-11-01T00:21:14.105 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-11-01T00:21:14.105 DEBUG:teuthology.orchestra.run.smithi072:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-11-01T00:21:14.183 INFO:teuthology.orchestra.run.smithi072.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-11-01T00:21:14.188 INFO:tasks.ceph:Archiving crash dumps... 2023-11-01T00:21:14.189 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/lib/ceph/crash to /home/teuthworker/archive/matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi/7442572/remote/smithi072/crash 2023-11-01T00:21:14.190 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-11-01T00:21:14.253 INFO:tasks.ceph:Compressing logs... 2023-11-01T00:21:14.253 DEBUG:teuthology.orchestra.run.smithi072:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-01T00:21:14.323 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-11-01T00:21:14.324 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83312.log 2023-11-01T00:21:14.324 INFO:teuthology.orchestra.run.smithi072.stderr:gzip/var/log/ceph/ceph-mon.a.log: -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.79328.log 2023-11-01T00:21:14.324 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81720.log 2023-11-01T00:21:14.326 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.83312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81972.log 2023-11-01T00:21:14.326 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph.tmp-client.admin.79328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80886.log 2023-11-01T00:21:14.327 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80926.log 2023-11-01T00:21:14.327 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83225.log 2023-11-01T00:21:14.327 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-11-01T00:21:14.327 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82224.log 2023-11-01T00:21:14.328 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82140.log 2023-11-01T00:21:14.328 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81659.log 2023-11-01T00:21:14.328 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82182.log 2023-11-01T00:21:14.328 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-11-01T00:21:14.328 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83750.log 2023-11-01T00:21:14.329 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82012.log 2023-11-01T00:21:14.329 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81561.log 2023-11-01T00:21:14.329 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80966.log 2023-11-01T00:21:14.333 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81762.log 2023-11-01T00:21:14.333 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81154.log 2023-11-01T00:21:14.333 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80776.log 2023-11-01T00:21:14.333 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81886.log 2023-11-01T00:21:14.333 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-11-01T00:21:14.334 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82098.log 2023-11-01T00:21:14.334 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81124.log 2023-11-01T00:21:14.334 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83376.log 2023-11-01T00:21:14.334 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83265.log 2023-11-01T00:21:14.334 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82011.log 2023-11-01T00:21:14.334 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-11-01T00:21:14.335 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80688.log 2023-11-01T00:21:14.335 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81617.log 2023-11-01T00:21:14.335 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81885.log 2023-11-01T00:21:14.335 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-11-01T00:21:14.335 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-11-01T00:21:14.336 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83288.log 2023-11-01T00:21:14.336 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83377.log 2023-11-01T00:21:14.336 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81884.log 2023-11-01T00:21:14.336 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82266.log 2023-11-01T00:21:14.336 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83162.log 2023-11-01T00:21:14.336 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81804.log 2023-11-01T00:21:14.365 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.81720.log: /var/log/ceph/ceph-client.admin.80886.log: /var/log/ceph/ceph-client.admin.80776.log: /var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.82098.log: /var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.83162.log: /var/log/ceph/ceph-client.admin.83750.log: /var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.83376.log: /var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph.log: /var/log/ceph/ceph-osd.2.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83750.log.gz 2023-11-01T00:21:14.366 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.83377.log: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-11-01T00:21:14.366 INFO:teuthology.orchestra.run.smithi072.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.79328.log.gz 2023-11-01T00:21:14.366 INFO:teuthology.orchestra.run.smithi072.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.83376.log.gz/var/log/ceph/ceph-client.admin.81972.log: 2023-11-01T00:21:14.366 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.83225.log: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.83377.log.gz 2023-11-01T00:21:14.367 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.82182.log: /var/log/ceph/ceph-client.admin.81659.log: /var/log/ceph/ceph-client.admin.80926.log: /var/log/ceph/ceph-client.admin.82140.log: /var/log/ceph/ceph-client.admin.82224.log: /var/log/ceph/ceph-client.admin.82011.log: /var/log/ceph/ceph-client.admin.80688.log: /var/log/ceph/ceph-client.admin.81561.log: /var/log/ceph/ceph-client.admin.80966.log: /var/log/ceph/ceph-client.admin.83265.log: /var/log/ceph/ceph-client.admin.81124.log: /var/log/ceph/ceph-client.admin.82012.log: 85.0% -- replaced with /var/log/ceph/ceph.log.gz 2023-11-01T00:21:14.370 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/ceph-client.admin.82266.log: /var/log/ceph/ceph-client.admin.81885.log: /var/log/ceph/ceph-client.admin.81762.log: /var/log/ceph/ceph-client.admin.81617.log: /var/log/ceph/ceph-client.admin.83288.log: /var/log/ceph/ceph-client.admin.81154.log: /var/log/ceph/ceph-client.admin.81804.log: /var/log/ceph/ceph-client.admin.81884.log: /var/log/ceph/ceph-client.admin.81886.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83312.log.gz 2023-11-01T00:21:14.373 INFO:teuthology.orchestra.run.smithi072.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80776.log.gz 2023-11-01T00:21:14.374 INFO:teuthology.orchestra.run.smithi072.stderr: 87.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-11-01T00:21:14.379 INFO:teuthology.orchestra.run.smithi072.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82098.log.gz 2023-11-01T00:21:14.379 INFO:teuthology.orchestra.run.smithi072.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.82140.log.gz 2023-11-01T00:21:14.379 INFO:teuthology.orchestra.run.smithi072.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82182.log.gz 2023-11-01T00:21:14.379 INFO:teuthology.orchestra.run.smithi072.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82224.log.gz 2023-11-01T00:21:14.379 INFO:teuthology.orchestra.run.smithi072.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82266.log.gz 2023-11-01T00:21:14.382 INFO:teuthology.orchestra.run.smithi072.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80886.log.gz 2023-11-01T00:21:14.385 INFO:teuthology.orchestra.run.smithi072.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.80966.log.gz 2023-11-01T00:21:14.389 INFO:teuthology.orchestra.run.smithi072.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81124.log.gz 2023-11-01T00:21:14.390 INFO:teuthology.orchestra.run.smithi072.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.80926.log.gz 2023-11-01T00:21:14.390 INFO:teuthology.orchestra.run.smithi072.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81561.log.gz 2023-11-01T00:21:14.390 INFO:teuthology.orchestra.run.smithi072.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81617.log.gz 2023-11-01T00:21:14.390 INFO:teuthology.orchestra.run.smithi072.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.81659.log.gz 2023-11-01T00:21:14.390 INFO:teuthology.orchestra.run.smithi072.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81720.log.gz 2023-11-01T00:21:14.391 INFO:teuthology.orchestra.run.smithi072.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81762.log.gz 2023-11-01T00:21:14.391 INFO:teuthology.orchestra.run.smithi072.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81804.log.gz 2023-11-01T00:21:14.391 INFO:teuthology.orchestra.run.smithi072.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81885.log.gz 2023-11-01T00:21:14.391 INFO:teuthology.orchestra.run.smithi072.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.81886.log.gz 2023-11-01T00:21:14.391 INFO:teuthology.orchestra.run.smithi072.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.81884.log.gz 2023-11-01T00:21:14.391 INFO:teuthology.orchestra.run.smithi072.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.81972.log.gz 2023-11-01T00:21:14.392 INFO:teuthology.orchestra.run.smithi072.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.82011.log.gz 2023-11-01T00:21:14.392 INFO:teuthology.orchestra.run.smithi072.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.82012.log.gz 2023-11-01T00:21:14.406 INFO:teuthology.orchestra.run.smithi072.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83162.log.gz 2023-11-01T00:21:14.406 INFO:teuthology.orchestra.run.smithi072.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83225.log.gz 2023-11-01T00:21:14.407 INFO:teuthology.orchestra.run.smithi072.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.83265.log.gz 2023-11-01T00:21:14.407 INFO:teuthology.orchestra.run.smithi072.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.80688.log.gz 2023-11-01T00:21:14.443 INFO:teuthology.orchestra.run.smithi072.stderr: 50.4% 50.4% -- replaced with /var/log/ceph/ceph-client.admin.83288.log.gz -- replaced with /var/log/ceph/ceph-client.admin.81154.log.gz 2023-11-01T00:21:14.443 INFO:teuthology.orchestra.run.smithi072.stderr: 2023-11-01T00:21:14.531 INFO:teuthology.orchestra.run.smithi072.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-11-01T00:21:16.396 INFO:teuthology.orchestra.run.smithi072.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-11-01T00:21:16.405 INFO:teuthology.orchestra.run.smithi072.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-11-01T00:21:16.518 INFO:teuthology.orchestra.run.smithi072.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-11-01T00:21:16.520 INFO:teuthology.orchestra.run.smithi072.stderr: 2023-11-01T00:21:16.521 INFO:teuthology.orchestra.run.smithi072.stderr:real 0m2.213s 2023-11-01T00:21:16.521 INFO:teuthology.orchestra.run.smithi072.stderr:user 0m3.815s 2023-11-01T00:21:16.521 INFO:teuthology.orchestra.run.smithi072.stderr:sys 0m0.179s 2023-11-01T00:21:16.521 INFO:tasks.ceph:Archiving logs... 2023-11-01T00:21:16.522 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/log/ceph to /home/teuthworker/archive/matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi/7442572/remote/smithi072/log 2023-11-01T00:21:16.523 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-01T00:21:17.125 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-01T00:21:17.135 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-01T00:21:17.135 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-01T00:21:17.194 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-01T00:21:17.202 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-01T00:21:17.203 DEBUG:teuthology.orchestra.run.smithi072:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-01T00:21:17.224 INFO:teuthology.orchestra.run.smithi072.stderr:bash: line 1: ntpq: command not found 2023-11-01T00:21:17.301 INFO:teuthology.orchestra.run.smithi072.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-01T00:21:17.301 INFO:teuthology.orchestra.run.smithi072.stdout:=============================================================================== 2023-11-01T00:21:17.301 INFO:teuthology.orchestra.run.smithi072.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 14 +762us[ +768us] +/- 79ms 2023-11-01T00:21:17.302 INFO:teuthology.orchestra.run.smithi072.stdout:^* hv02.front.sepia.ceph.com 3 6 377 14 -435us[ -431us] +/- 51ms 2023-11-01T00:21:17.302 INFO:teuthology.orchestra.run.smithi072.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 14 -75us[ -75us] +/- 65ms 2023-11-01T00:21:17.302 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-01T00:21:17.327 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-01T00:21:17.336 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-01T00:21:17.336 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-01T00:21:17.346 DEBUG:teuthology.orchestra.run.smithi072:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-01T00:21:17.569 DEBUG:teuthology.orchestra.run.smithi072:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-01T00:21:17.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T00:21:17.600 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-01T00:21:17.609 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-01T00:21:17.617 INFO:teuthology.task.internal:Duration was 734.373125 seconds 2023-11-01T00:21:17.617 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-01T00:21:17.625 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-01T00:21:17.625 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-01T00:21:17.797 INFO:teuthology.orchestra.run.smithi072.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-01T00:21:18.208 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-01T00:21:18.209 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi072.front.sepia.ceph.com 2023-11-01T00:21:18.209 DEBUG:teuthology.orchestra.run.smithi072:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-01T00:21:18.229 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-01T00:21:18.229 DEBUG:teuthology.orchestra.run.smithi072:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-01T00:21:18.298 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-01T00:21:18.298 DEBUG:teuthology.orchestra.run.smithi072:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-01T00:21:18.429 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-01T00:21:18.439 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-01T00:21:18.439 DEBUG:teuthology.orchestra.run.smithi072:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-01T00:21:18.473 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-01T00:21:18.482 DEBUG:teuthology.orchestra.run.smithi072:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-01T00:21:18.556 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern = core 2023-11-01T00:21:18.578 DEBUG:teuthology.orchestra.run.smithi072:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-01T00:21:18.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-01T00:21:18.632 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-01T00:21:18.642 INFO:teuthology.task.internal:Transferring archived files... 2023-11-01T00:21:18.642 DEBUG:teuthology.misc:Transferring archived files from smithi072:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi/7442572/remote/smithi072 2023-11-01T00:21:18.643 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-01T00:21:18.741 INFO:teuthology.task.internal:Removing archive directory... 2023-11-01T00:21:18.741 DEBUG:teuthology.orchestra.run.smithi072:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-01T00:21:18.785 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-01T00:21:18.797 INFO:teuthology.task.internal:Not uploading archives. 2023-11-01T00:21:18.798 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-01T00:21:18.807 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-01T00:21:18.807 DEBUG:teuthology.orchestra.run.smithi072:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-01T00:21:18.839 INFO:teuthology.orchestra.run.smithi072.stdout: 265465 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 1 00:21 /home/ubuntu/cephtest 2023-11-01T00:21:18.840 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-01T00:21:18.850 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-01T00:21:18.871 INFO:teuthology.nuke:Checking targets against current locks 2023-11-01T00:21:18.898 DEBUG:teuthology.nuke:shortname: smithi072 2023-11-01T00:21:18.898 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-01T00:21:18.919 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi072.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/matan-2023-10-31_22:58:07-crimson-rados-wip-matanb-crimson-do_osd_ops_execute-part3-distro-crimson-smithi/7442572', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-01 00:00:00.606917', 'locked_by': 'scheduled_matan@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPqQ8m1lyC4gfmIhCRK9ue+TQpRlEZDrwDAYyVhYmeVhgru8v0m8+bPoyhVsLbnuMQxri2+3S78DUdW7KMv+nhg='} 2023-11-01T00:21:18.965 INFO:teuthology.orchestra.console.smithi072:Power off 2023-11-01T00:21:18.966 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-01T00:21:18.990 DEBUG:teuthology.orchestra.console.smithi072:power off output: Chassis Power Control: Down/Off 2023-11-01T00:21:18.990 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-01T00:21:19.012 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2023-11-01T00:21:23.015 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-01T00:21:23.124 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2023-11-01T00:21:27.129 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-01T00:21:27.238 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is off 2023-11-01T00:21:27.339 INFO:teuthology.orchestra.console.smithi072:Power off completed 2023-11-01T00:21:27.442 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/Rfy7LkB4k/rados-bench?orgId=1&var-branch_name=wip-matanb-crimson-do_osd_ops_execute-part3&var-job_id_selected=7442572 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} duration: 734.373125076294 failure_reason: '"2023-11-01T00:20:32.740993+0000 mon.a (mon.0) 155 : cluster [WRN] Health check failed: Degraded data redundancy: 11/1808 objects degraded (0.608%), 2 pgs degraded (PG_DEGRADED)" in cluster log' flavor: crimson owner: scheduled_matan@teuthology success: false 2023-11-01T00:21:27.443 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-01T00:21:27.529 INFO:teuthology.run:FAIL