2024-01-19T15:40:59.265 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-19T15:40:59.265 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yingxin@teuthology --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} --name yingxin-2024-01-19_07:32:18-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-8-distro-default-smithi --archive /home/teuthworker/archive/yingxin-2024-01-19_07:32:18-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-8-distro-default-smithi/7521657 --verbose -- /home/teuthworker/archive/yingxin-2024-01-19_07:32:18-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-8-distro-default-smithi/7521657/orig.config.yaml 2024-01-19T15:40:59.308 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-19T15:40:59.402 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yingxin-2024-01-19_07:32:18-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-8-distro-default-smithi/7521657 branch: wip-yingxin-crimson-save-conn-foreign-copy-8 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} email: yingxin.cheng@intel.com first_in_suite: false job_id: '7521657' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yingxin-2024-01-19_07:32:18-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-8-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yingxin-crimson-save-conn-foreign-copy-8 ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson osd obc lru size: 10 crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4 workunit: branch: wip-yingxin-crimson-save-conn-foreign-copy-8 env: CRIMSON_COMPAT: '1' sha1: bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4 owner: scheduled_yingxin@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4 sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-yingxin-crimson-save-conn-foreign-copy-8 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4 targets: smithi046.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIEuHVGCLxJS4bLidw0RCCIWGTwemGpysryduvv9ZtUC9CsMI8rzNHbsqgtTiTlXeWgF/Tr87RV8+hsaqzDi3Mw= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: librbdfio: iodepth: - 32 log_avg_msec: 100 mode: - randread norandommap: true op_size: - 4096 osd_ra: - 4096 pool_profile: rbd procs_per_volume: - 1 time: 60 vol_size: 4096 volumes_per_client: - 2 cluster: iterations: 1 osds_per_node: 3 pool_profiles: rbd: pg_size: 128 pgp_size: 128 replication: 3 user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-19_07:32:18 tube: smithi user: yingxin verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-19T15:40:59.402 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4/qa; will attempt to use it 2024-01-19T15:40:59.403 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4/qa/tasks 2024-01-19T15:40:59.403 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-19T15:40:59.405 INFO:teuthology.task.internal:Checking packages... 2024-01-19T15:40:59.427 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash 'bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4' 2024-01-19T15:40:59.427 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-19T15:40:59.427 INFO:teuthology.packaging:ref: None 2024-01-19T15:40:59.427 INFO:teuthology.packaging:tag: None 2024-01-19T15:40:59.427 INFO:teuthology.packaging:branch: wip-yingxin-crimson-save-conn-foreign-copy-8 2024-01-19T15:40:59.427 INFO:teuthology.packaging:sha1: bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4 2024-01-19T15:40:59.428 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-yingxin-crimson-save-conn-foreign-copy-8 2024-01-19T15:40:59.651 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-767.gbdd6070c 2024-01-19T15:40:59.652 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-19T15:40:59.659 INFO:teuthology.task.internal:no buildpackages task found 2024-01-19T15:40:59.659 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-19T15:40:59.685 INFO:teuthology.task.internal:Saving configuration 2024-01-19T15:40:59.700 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-19T15:40:59.706 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-19T15:40:59.730 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-01-19_07:32:18-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-8-distro-default-smithi/7521657', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-19 15:36:00.828024', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIEuHVGCLxJS4bLidw0RCCIWGTwemGpysryduvv9ZtUC9CsMI8rzNHbsqgtTiTlXeWgF/Tr87RV8+hsaqzDi3Mw='} 2024-01-19T15:40:59.730 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-19T15:40:59.735 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-01-19T15:40:59.735 INFO:teuthology.run_tasks:Running task console_log... 2024-01-19T15:40:59.772 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f538a5d0dc0>, signals=[15]) 2024-01-19T15:40:59.772 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-19T15:40:59.778 INFO:teuthology.task.internal:Opening connections... 2024-01-19T15:40:59.778 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:40:59.780 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T15:40:59.883 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-19T15:40:59.889 DEBUG:teuthology.orchestra.run.smithi046:> uname -m 2024-01-19T15:40:59.953 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64 2024-01-19T15:40:59.953 DEBUG:teuthology.orchestra.run.smithi046:> cat /etc/os-release 2024-01-19T15:41:00.010 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="CentOS Stream" 2024-01-19T15:41:00.010 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="9" 2024-01-19T15:41:00.010 INFO:teuthology.orchestra.run.smithi046.stdout:ID="centos" 2024-01-19T15:41:00.010 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE="rhel fedora" 2024-01-19T15:41:00.010 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="9" 2024-01-19T15:41:00.010 INFO:teuthology.orchestra.run.smithi046.stdout:PLATFORM_ID="platform:el9" 2024-01-19T15:41:00.010 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-19T15:41:00.010 INFO:teuthology.orchestra.run.smithi046.stdout:ANSI_COLOR="0;31" 2024-01-19T15:41:00.011 INFO:teuthology.orchestra.run.smithi046.stdout:LOGO="fedora-logo-icon" 2024-01-19T15:41:00.011 INFO:teuthology.orchestra.run.smithi046.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-19T15:41:00.011 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://centos.org/" 2024-01-19T15:41:00.011 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-19T15:41:00.011 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-19T15:41:00.011 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-19T15:41:00.011 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2024-01-19T15:41:00.041 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-19T15:41:00.049 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-19T15:41:00.054 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-19T15:41:00.055 DEBUG:teuthology.orchestra.run.smithi046:> test '!' -e /home/ubuntu/cephtest 2024-01-19T15:41:00.071 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-19T15:41:00.077 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-19T15:41:00.077 DEBUG:teuthology.orchestra.run.smithi046:> test -z $(ls -A /var/lib/ceph) 2024-01-19T15:41:00.140 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-19T15:41:00.167 INFO:teuthology.run_tasks:Running task kernel... 2024-01-19T15:41:00.181 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-19T15:41:00.181 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-19T15:41:00.181 DEBUG:teuthology.orchestra.run.smithi046:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-19T15:41:00.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-19T15:41:00.198 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-01-19T15:41:00.256 INFO:teuthology.orchestra.run.smithi046.stdout:5.14.0-361.el9.x86_64 2024-01-19T15:41:00.256 INFO:teuthology.task.kernel:Running kernel on smithi046: 5.14.0-361.el9.x86_64 2024-01-19T15:41:00.256 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum install -y kernel 2024-01-19T15:41:02.393 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - AppStream 13 MB/s | 6.4 MB 00:00 2024-01-19T15:41:03.760 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-01-19T15:41:04.875 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-01-19T15:41:11.718 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - AppStream 3.2 MB/s | 19 MB 00:05 2024-01-19T15:41:16.543 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - CRB 4.7 MB/s | 5.9 MB 00:01 2024-01-19T15:41:18.902 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 15 kB 00:01 2024-01-19T15:41:28.821 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 2.1 MB/s | 20 MB 00:09 2024-01-19T15:41:33.944 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-01-19T15:41:34.136 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:01 ago on Fri 19 Jan 2024 03:41:33 PM UTC. 2024-01-19T15:41:36.235 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-01-19T15:41:36.235 INFO:teuthology.orchestra.run.smithi046.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-01-19T15:41:36.426 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:41:36.430 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:41:36.430 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-01-19T15:41:36.430 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:41:36.430 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-19T15:41:36.430 INFO:teuthology.orchestra.run.smithi046.stdout: kernel x86_64 5.14.0-409.el9 CentOS-BaseOS 6.0 M 2024-01-19T15:41:36.430 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-01-19T15:41:36.431 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-core x86_64 5.14.0-409.el9 CentOS-BaseOS 21 M 2024-01-19T15:41:36.431 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules x86_64 5.14.0-409.el9 CentOS-BaseOS 39 M 2024-01-19T15:41:36.431 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules-core x86_64 5.14.0-409.el9 CentOS-BaseOS 33 M 2024-01-19T15:41:36.431 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:41:36.431 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-19T15:41:36.431 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:41:36.431 INFO:teuthology.orchestra.run.smithi046.stdout:Install 4 Packages 2024-01-19T15:41:36.431 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:41:36.432 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 99 M 2024-01-19T15:41:36.432 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 122 M 2024-01-19T15:41:36.432 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-19T15:41:37.154 INFO:teuthology.orchestra.run.smithi046.stdout:(1/4): kernel-5.14.0-409.el9.x86_64.rpm 8.3 MB/s | 6.0 MB 00:00 2024-01-19T15:41:38.296 INFO:teuthology.orchestra.run.smithi046.stdout:(2/4): kernel-core-5.14.0-409.el9.x86_64.rpm 11 MB/s | 21 MB 00:01 2024-01-19T15:41:38.855 INFO:teuthology.orchestra.run.smithi046.stdout:(3/4): kernel-modules-5.14.0-409.el9.x86_64.rpm 16 MB/s | 39 MB 00:02 2024-01-19T15:41:39.154 INFO:teuthology.orchestra.run.smithi046.stdout:(4/4): kernel-modules-core-5.14.0-409.el9.x86_6 17 MB/s | 33 MB 00:01 2024-01-19T15:41:39.157 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:41:39.158 INFO:teuthology.orchestra.run.smithi046.stdout:Total 36 MB/s | 99 MB 00:02 2024-01-19T15:41:39.158 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-19T15:41:39.586 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-19T15:41:39.586 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-19T15:41:40.673 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-19T15:41:40.673 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-19T15:41:43.206 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-19T15:41:43.886 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-modules-core-5.14.0-409.el9.x86_64 1/4 2024-01-19T15:41:44.012 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-core-5.14.0-409.el9.x86_64 2/4 2024-01-19T15:41:46.899 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-core-5.14.0-409.el9.x86_64 2/4 2024-01-19T15:41:47.088 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-modules-5.14.0-409.el9.x86_64 3/4 2024-01-19T15:41:52.160 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-modules-5.14.0-409.el9.x86_64 3/4 2024-01-19T15:41:52.259 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : kernel-5.14.0-409.el9.x86_64 4/4 2024-01-19T15:41:57.343 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-modules-core-5.14.0-409.el9.x86_64 4/4 2024-01-19T15:42:28.146 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-core-5.14.0-409.el9.x86_64 4/4 2024-01-19T15:42:28.146 INFO:teuthology.orchestra.run.smithi046.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-19T15:42:28.146 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:42:28.219 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-modules-5.14.0-409.el9.x86_64 4/4 2024-01-19T15:42:28.602 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: kernel-5.14.0-409.el9.x86_64 4/4 2024-01-19T15:42:28.602 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-5.14.0-409.el9.x86_64 1/4 2024-01-19T15:42:28.602 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-core-5.14.0-409.el9.x86_64 2/4 2024-01-19T15:42:28.602 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-modules-5.14.0-409.el9.x86_64 3/4 2024-01-19T15:42:29.032 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : kernel-modules-core-5.14.0-409.el9.x86_64 4/4 2024-01-19T15:42:29.033 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:42:29.033 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-19T15:42:29.033 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-5.14.0-409.el9.x86_64 2024-01-19T15:42:29.033 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-core-5.14.0-409.el9.x86_64 2024-01-19T15:42:29.033 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules-5.14.0-409.el9.x86_64 2024-01-19T15:42:29.033 INFO:teuthology.orchestra.run.smithi046.stdout: kernel-modules-core-5.14.0-409.el9.x86_64 2024-01-19T15:42:29.033 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:42:29.033 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:42:29.564 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q kernel | sort -rV | head -n 1 2024-01-19T15:42:29.705 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-5.14.0-409.el9.x86_64 2024-01-19T15:42:29.705 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-409.el9.x86_64 2024-01-19T15:42:29.705 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-409.el9.x86_64 2024-01-19T15:42:29.706 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-19T15:42:29.706 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-19T15:42:29.706 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-01-19T15:42:29.706 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi046.front.sepia.ceph.com, path=None, version=distro) 2024-01-19T15:42:29.706 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q kernel | sort -rV | head -n 1 2024-01-19T15:42:29.818 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-5.14.0-409.el9.x86_64 2024-01-19T15:42:29.818 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-409.el9.x86_64 2024-01-19T15:42:29.819 DEBUG:teuthology.orchestra.run.smithi046:> sudo rpm -qi grub2-tools 2024-01-19T15:42:29.862 INFO:teuthology.orchestra.run.smithi046.stdout:Name : grub2-tools 2024-01-19T15:42:29.862 INFO:teuthology.orchestra.run.smithi046.stdout:Epoch : 1 2024-01-19T15:42:29.862 INFO:teuthology.orchestra.run.smithi046.stdout:Version : 2.06 2024-01-19T15:42:29.863 INFO:teuthology.orchestra.run.smithi046.stdout:Release : 68.el9 2024-01-19T15:42:29.863 INFO:teuthology.orchestra.run.smithi046.stdout:Architecture: x86_64 2024-01-19T15:42:29.863 INFO:teuthology.orchestra.run.smithi046.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-19T15:42:29.863 INFO:teuthology.orchestra.run.smithi046.stdout:Group : Unspecified 2024-01-19T15:42:29.863 INFO:teuthology.orchestra.run.smithi046.stdout:Size : 8264254 2024-01-19T15:42:29.863 INFO:teuthology.orchestra.run.smithi046.stdout:License : GPLv3+ 2024-01-19T15:42:29.863 INFO:teuthology.orchestra.run.smithi046.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-19T15:42:29.863 INFO:teuthology.orchestra.run.smithi046.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-19T15:42:29.863 INFO:teuthology.orchestra.run.smithi046.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-19T15:42:29.863 INFO:teuthology.orchestra.run.smithi046.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-19T15:42:29.863 INFO:teuthology.orchestra.run.smithi046.stdout:Packager : builder@centos.org 2024-01-19T15:42:29.863 INFO:teuthology.orchestra.run.smithi046.stdout:Vendor : CentOS 2024-01-19T15:42:29.863 INFO:teuthology.orchestra.run.smithi046.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-19T15:42:29.863 INFO:teuthology.orchestra.run.smithi046.stdout:Summary : Support tools for GRUB. 2024-01-19T15:42:29.864 INFO:teuthology.orchestra.run.smithi046.stdout:Description : 2024-01-19T15:42:29.864 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:42:29.864 INFO:teuthology.orchestra.run.smithi046.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-19T15:42:29.864 INFO:teuthology.orchestra.run.smithi046.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-19T15:42:29.864 INFO:teuthology.orchestra.run.smithi046.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-19T15:42:29.864 INFO:teuthology.orchestra.run.smithi046.stdout:hardware devices. 2024-01-19T15:42:29.864 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:42:29.864 INFO:teuthology.orchestra.run.smithi046.stdout:This subpackage provides tools for support of all platforms. 2024-01-19T15:42:29.865 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-19T15:42:29.865 INFO:teuthology.task.kernel:Updating grub on smithi046 to boot 5.14.0-409.el9.x86_64 2024-01-19T15:42:29.865 DEBUG:teuthology.orchestra.run.smithi046:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-19T15:42:30.447 INFO:teuthology.orchestra.run.smithi046.stderr:Generating grub configuration file ... 2024-01-19T15:42:32.033 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[32352]) leaked on vgs invocation. Parent PID 18142: grub2-probe 2024-01-19T15:42:32.033 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[41728]) leaked on vgs invocation. Parent PID 18142: grub2-probe 2024-01-19T15:42:32.063 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[32352]) leaked on vgs invocation. Parent PID 18142: grub2-probe 2024-01-19T15:42:32.063 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[41728]) leaked on vgs invocation. Parent PID 18142: grub2-probe 2024-01-19T15:42:32.347 INFO:teuthology.orchestra.run.smithi046.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-19T15:42:32.389 INFO:teuthology.orchestra.run.smithi046.stderr:done 2024-01-19T15:42:32.391 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2024-01-19T15:42:32.406 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.VvHdZm41SG 2024-01-19T15:42:32.407 DEBUG:teuthology.orchestra.run.smithi046:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.VvHdZm41SG 2024-01-19T15:42:32.486 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.VvHdZm41SG 2024-01-19T15:42:32.633 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.VvHdZm41SG is 6KB 2024-01-19T15:42:32.682 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.VvHdZm41SG 2024-01-19T15:42:32.697 DEBUG:teuthology.orchestra.run.smithi046:> sudo /bin/ls /boot/loader/entries || true 2024-01-19T15:42:32.776 INFO:teuthology.orchestra.run.smithi046.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-19T15:42:32.776 INFO:teuthology.orchestra.run.smithi046.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-19T15:42:32.776 INFO:teuthology.orchestra.run.smithi046.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-19T15:42:32.776 INFO:teuthology.orchestra.run.smithi046.stdout:15949e3c88704300882482eddfd7baae-5.14.0-409.el9.x86_64.conf 2024-01-19T15:42:32.778 DEBUG:teuthology.orchestra.run.smithi046:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-409.el9.x86_64 2024-01-19T15:42:32.937 DEBUG:teuthology.orchestra.run.smithi046:> sudo shutdown -r now 2024-01-19T15:43:02.956 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-01-19T15:43:02.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-01-19T15:43:02.958 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T15:43:37.532 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2024-01-19T15:43:46.540 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-01-19T15:43:46.541 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T15:44:04.924 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2024-01-19T15:44:16.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-01-19T15:44:16.935 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T15:44:17.242 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-01-19T15:44:17.794 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-01-19T15:44:17.794 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-409.el9.x86_64"... 2024-01-19T15:44:17.795 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-01-19T15:44:17.808 INFO:teuthology.orchestra.run.smithi046.stdout:5.14.0-409.el9.x86_64 2024-01-19T15:44:17.809 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-409.el9.x86_64 vs 5.14.0-409.el9.x86_64 2024-01-19T15:44:17.809 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-19T15:44:17.809 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-19T15:44:18.809 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-01-19T15:44:18.810 DEBUG:teuthology.orchestra.run.smithi046:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-19T15:44:18.881 INFO:teuthology.orchestra.run.smithi046.stdout:ttyS1 2024-01-19T15:44:18.914 DEBUG:teuthology.parallel:result is None 2024-01-19T15:44:18.914 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-19T15:44:18.923 INFO:teuthology.task.internal:Creating test directory... 2024-01-19T15:44:18.923 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-19T15:44:18.937 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-19T15:44:18.945 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-19T15:44:18.953 INFO:teuthology.task.internal:Creating archive directory... 2024-01-19T15:44:18.953 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-19T15:44:19.007 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-19T15:44:19.014 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-19T15:44:19.015 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-19T15:44:19.092 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-19T15:44:19.104 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-19T15:44:19.106 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-19T15:44:19.114 INFO:teuthology.task.internal:Configuring sudo... 2024-01-19T15:44:19.115 DEBUG:teuthology.orchestra.run.smithi046:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-19T15:44:19.142 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-19T15:44:19.153 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-19T15:44:19.153 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-19T15:44:19.198 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-19T15:44:19.280 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-19T15:44:19.328 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-19T15:44:19.406 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-19T15:44:19.454 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-19T15:44:19.454 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-19T15:44:19.530 DEBUG:teuthology.orchestra.run.smithi046:> sudo service rsyslog restart 2024-01-19T15:44:19.632 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-19T15:44:19.932 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-19T15:44:19.942 INFO:teuthology.task.internal:Starting timer... 2024-01-19T15:44:19.942 INFO:teuthology.run_tasks:Running task pcp... 2024-01-19T15:44:19.959 INFO:teuthology.run_tasks:Running task selinux... 2024-01-19T15:44:19.975 DEBUG:teuthology.orchestra.run.smithi046:> sudo service auditd rotate 2024-01-19T15:44:20.050 INFO:teuthology.orchestra.run.smithi046.stdout:Rotating logs: 2024-01-19T15:44:20.051 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-19T15:44:20.052 DEBUG:teuthology.orchestra.run.smithi046:> /usr/sbin/getenforce 2024-01-19T15:44:20.070 INFO:teuthology.orchestra.run.smithi046.stdout:Permissive 2024-01-19T15:44:20.070 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi046.front.sepia.ceph.com': 'permissive'} 2024-01-19T15:44:20.071 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-19T15:44:20.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-19T15:44:20.139 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-19T15:44:20.139 DEBUG:teuthology.orchestra.run.smithi046:> sudo /usr/sbin/setenforce permissive 2024-01-19T15:44:20.215 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-19T15:44:20.226 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-19T15:44:20.243 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-19T15:44:20.245 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi046.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-19T15:48:32.990 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi046.front.sepia.ceph.com')] 2024-01-19T15:48:32.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-01-19T15:48:32.992 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T15:48:33.093 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-01-19T15:48:33.172 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-01-19T15:48:33.172 INFO:teuthology.run_tasks:Running task clock... 2024-01-19T15:48:33.183 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-19T15:48:33.183 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-19T15:48:33.184 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-19T15:48:33.245 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-19T15:48:33.262 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-19T15:48:33.292 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: ntpd: command not found 2024-01-19T15:48:33.306 INFO:teuthology.orchestra.run.smithi046.stdout:506 Cannot talk to daemon 2024-01-19T15:48:33.322 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-19T15:48:33.339 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-19T15:48:33.399 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: ntpq: command not found 2024-01-19T15:48:33.402 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-19T15:48:33.402 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-01-19T15:48:33.402 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T15:48:33.402 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T15:48:33.402 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T15:48:33.402 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T15:48:33.403 INFO:teuthology.run_tasks:Running task install... 2024-01-19T15:48:33.412 DEBUG:teuthology.task.install:project ceph 2024-01-19T15:48:33.412 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': 'bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4'}} 2024-01-19T15:48:33.412 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': 'bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4'} 2024-01-19T15:48:33.412 INFO:teuthology.task.install:Using flavor: crimson 2024-01-19T15:48:33.418 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-19T15:48:33.418 INFO:teuthology.task.install:extra packages: [] 2024-01-19T15:48:33.418 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4', 'tag': None, 'wait_for_package': False} 2024-01-19T15:48:33.419 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4 2024-01-19T15:48:33.773 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yingxin-crimson-save-conn-foreign-copy-8/bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4/centos/9/flavors/crimson/ 2024-01-19T15:48:33.774 INFO:teuthology.task.install.rpm:Package version is 19.0.0-767.gbdd6070c 2024-01-19T15:48:34.005 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yingxin-crimson-save-conn-foreign-copy-8/bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yingxin-crimson-save-conn-foreign-copy-8/bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yingxin-crimson-save-conn-foreign-copy-8/bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-19T15:48:34.005 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-19T15:48:34.005 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-19T15:48:34.036 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-19T15:48:34.037 DEBUG:teuthology.orchestra.run.smithi046:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-19T15:48:34.106 DEBUG:teuthology.orchestra.run.smithi046:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-19T15:48:34.188 DEBUG:teuthology.orchestra.run.smithi046:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-19T15:48:34.255 INFO:teuthology.orchestra.run.smithi046.stdout:check_obsoletes = 1 2024-01-19T15:48:34.257 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum clean all 2024-01-19T15:48:34.503 INFO:teuthology.orchestra.run.smithi046.stdout:73 files removed 2024-01-19T15:48:34.530 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-radosgw 2024-01-19T15:48:35.206 INFO:teuthology.orchestra.run.smithi046.stdout:ceph packages for x86_64 237 kB/s | 83 kB 00:00 2024-01-19T15:48:35.615 INFO:teuthology.orchestra.run.smithi046.stdout:ceph noarch packages 27 kB/s | 9.5 kB 00:00 2024-01-19T15:48:36.049 INFO:teuthology.orchestra.run.smithi046.stdout:ceph source packages 5.0 kB/s | 2.0 kB 00:00 2024-01-19T15:48:36.634 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - AppStream 12 MB/s | 6.4 MB 00:00 2024-01-19T15:48:38.043 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-01-19T15:48:39.076 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-01-19T15:48:41.302 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-01-19T15:48:46.764 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 5.9 MB 00:01 2024-01-19T15:48:48.278 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 15 kB 00:00 2024-01-19T15:48:55.557 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 2.8 MB/s | 20 MB 00:07 2024-01-19T15:49:00.604 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-01-19T15:49:03.200 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:49:03.215 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:49:03.215 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-01-19T15:49:03.216 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:49:03.216 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-19T15:49:03.216 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 8.4 M 2024-01-19T15:49:03.216 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrading: 2024-01-19T15:49:03.216 INFO:teuthology.orchestra.run.smithi046.stdout: librados2 x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 3.1 M 2024-01-19T15:49:03.216 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1 x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 3.1 M 2024-01-19T15:49:03.216 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-01-19T15:49:03.216 INFO:teuthology.orchestra.run.smithi046.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-19T15:49:03.216 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 5.1 M 2024-01-19T15:49:03.216 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 18 M 2024-01-19T15:49:03.216 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 25 k 2024-01-19T15:49:03.216 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-19T15:49:03.217 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-19T15:49:03.217 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-19T15:49:03.217 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2 x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 752 k 2024-01-19T15:49:03.217 INFO:teuthology.orchestra.run.smithi046.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-19T15:49:03.217 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-19T15:49:03.217 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-19T15:49:03.217 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1 x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 496 k 2024-01-19T15:49:03.217 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-19T15:49:03.217 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2 x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 4.5 M 2024-01-19T15:49:03.217 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-19T15:49:03.217 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-19T15:49:03.217 INFO:teuthology.orchestra.run.smithi046.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-19T15:49:03.217 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 45 k 2024-01-19T15:49:03.217 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 130 k 2024-01-19T15:49:03.218 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 162 k 2024-01-19T15:49:03.218 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-19T15:49:03.218 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 320 k 2024-01-19T15:49:03.218 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 298 k 2024-01-19T15:49:03.218 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 99 k 2024-01-19T15:49:03.218 INFO:teuthology.orchestra.run.smithi046.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-19T15:49:03.218 INFO:teuthology.orchestra.run.smithi046.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-19T15:49:03.218 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:49:03.218 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-19T15:49:03.218 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:49:03.218 INFO:teuthology.orchestra.run.smithi046.stdout:Install 27 Packages 2024-01-19T15:49:03.218 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrade 2 Packages 2024-01-19T15:49:03.218 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:49:03.219 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 53 M 2024-01-19T15:49:03.219 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-19T15:49:04.924 INFO:teuthology.orchestra.run.smithi046.stdout:(1/29): ceph-base-19.0.0-767.gbdd6070c.el9.x86_ 3.4 MB/s | 5.1 MB 00:01 2024-01-19T15:49:04.981 INFO:teuthology.orchestra.run.smithi046.stdout:(2/29): ceph-selinux-19.0.0-767.gbdd6070c.el9.x 449 kB/s | 25 kB 00:00 2024-01-19T15:49:05.139 INFO:teuthology.orchestra.run.smithi046.stdout:(3/29): libcephfs2-19.0.0-767.gbdd6070c.el9.x86 4.6 MB/s | 752 kB 00:00 2024-01-19T15:49:05.289 INFO:teuthology.orchestra.run.smithi046.stdout:(4/29): libradosstriper1-19.0.0-767.gbdd6070c.e 3.2 MB/s | 496 kB 00:00 2024-01-19T15:49:06.098 INFO:teuthology.orchestra.run.smithi046.stdout:(5/29): librgw2-19.0.0-767.gbdd6070c.el9.x86_64 5.6 MB/s | 4.5 MB 00:00 2024-01-19T15:49:06.198 INFO:teuthology.orchestra.run.smithi046.stdout:(6/29): ceph-radosgw-19.0.0-767.gbdd6070c.el9.x 3.0 MB/s | 8.4 MB 00:02 2024-01-19T15:49:06.223 INFO:teuthology.orchestra.run.smithi046.stdout:(7/29): python3-ceph-argparse-19.0.0-767.gbdd60 361 kB/s | 45 kB 00:00 2024-01-19T15:49:06.274 INFO:teuthology.orchestra.run.smithi046.stdout:(8/29): python3-ceph-common-19.0.0-767.gbdd6070 1.7 MB/s | 130 kB 00:00 2024-01-19T15:49:06.299 INFO:teuthology.orchestra.run.smithi046.stdout:(9/29): python3-cephfs-19.0.0-767.gbdd6070c.el9 2.1 MB/s | 162 kB 00:00 2024-01-19T15:49:06.391 INFO:teuthology.orchestra.run.smithi046.stdout:(10/29): python3-rbd-19.0.0-767.gbdd6070c.el9.x 3.2 MB/s | 298 kB 00:00 2024-01-19T15:49:06.416 INFO:teuthology.orchestra.run.smithi046.stdout:(11/29): python3-rados-19.0.0-767.gbdd6070c.el9 2.2 MB/s | 320 kB 00:00 2024-01-19T15:49:06.466 INFO:teuthology.orchestra.run.smithi046.stdout:(12/29): python3-rgw-19.0.0-767.gbdd6070c.el9.x 1.3 MB/s | 99 kB 00:00 2024-01-19T15:49:06.592 INFO:teuthology.orchestra.run.smithi046.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 604 kB/s | 106 kB 00:00 2024-01-19T15:49:06.617 INFO:teuthology.orchestra.run.smithi046.stdout:(14/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-01-19T15:49:06.642 INFO:teuthology.orchestra.run.smithi046.stdout:(15/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-01-19T15:49:06.701 INFO:teuthology.orchestra.run.smithi046.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 661 kB 00:00 2024-01-19T15:49:06.726 INFO:teuthology.orchestra.run.smithi046.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-01-19T15:49:06.768 INFO:teuthology.orchestra.run.smithi046.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 619 kB/s | 41 kB 00:00 2024-01-19T15:49:07.401 INFO:teuthology.orchestra.run.smithi046.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 457 kB/s | 309 kB 00:00 2024-01-19T15:49:07.510 INFO:teuthology.orchestra.run.smithi046.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-01-19T15:49:07.627 INFO:teuthology.orchestra.run.smithi046.stdout:(21/29): liboath-2.6.7-2.el9.x86_64.rpm 418 kB/s | 49 kB 00:00 2024-01-19T15:49:07.744 INFO:teuthology.orchestra.run.smithi046.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 582 kB/s | 67 kB 00:00 2024-01-19T15:49:07.811 INFO:teuthology.orchestra.run.smithi046.stdout:(23/29): libarrow-9.0.0-10.el9.x86_64.rpm 4.2 MB/s | 4.4 MB 00:01 2024-01-19T15:49:08.019 INFO:teuthology.orchestra.run.smithi046.stdout:(24/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.0 MB/s | 838 kB 00:00 2024-01-19T15:49:08.045 INFO:teuthology.orchestra.run.smithi046.stdout:(25/29): re2-20211101-3.el9.x86_64.rpm 821 kB/s | 192 kB 00:00 2024-01-19T15:49:08.278 INFO:teuthology.orchestra.run.smithi046.stdout:(26/29): thrift-0.15.0-2.el9.x86_64.rpm 6.1 MB/s | 1.6 MB 00:00 2024-01-19T15:49:08.837 INFO:teuthology.orchestra.run.smithi046.stdout:(27/29): librados2-19.0.0-767.gbdd6070c.el9.x86 4.0 MB/s | 3.1 MB 00:00 2024-01-19T15:49:09.529 INFO:teuthology.orchestra.run.smithi046.stdout:(28/29): librbd1-19.0.0-767.gbdd6070c.el9.x86_6 2.5 MB/s | 3.1 MB 00:01 2024-01-19T15:49:11.279 INFO:teuthology.orchestra.run.smithi046.stdout:(29/29): ceph-common-19.0.0-767.gbdd6070c.el9.x 2.2 MB/s | 18 MB 00:07 2024-01-19T15:49:11.285 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:49:11.285 INFO:teuthology.orchestra.run.smithi046.stdout:Total 6.5 MB/s | 53 MB 00:08 2024-01-19T15:49:11.286 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-19T15:49:11.326 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-19T15:49:11.326 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-19T15:49:11.786 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-19T15:49:11.786 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-19T15:49:13.172 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-19T15:49:13.563 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-01-19T15:49:13.631 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librados2-2:19.0.0-767.gbdd6070c.el9.x86_64 2/31 2024-01-19T15:49:13.848 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:19.0.0-767.gbdd6070c.el9.x86_64 2/31 2024-01-19T15:49:13.941 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rados-2:19.0.0-767.gbdd6070c.el9.x86_64 3/31 2024-01-19T15:49:13.974 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs2-2:19.0.0-767.gbdd6070c.el9.x86_64 4/31 2024-01-19T15:49:14.056 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephfs2-2:19.0.0-767.gbdd6070c.el9.x86_64 4/31 2024-01-19T15:49:14.086 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/31 2024-01-19T15:49:14.111 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/31 2024-01-19T15:49:14.141 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/31 2024-01-19T15:49:14.190 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-argparse-2:19.0.0-767.gbdd6070c.el9.x 8/31 2024-01-19T15:49:14.265 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cephfs-2:19.0.0-767.gbdd6070c.el9.x86_64 9/31 2024-01-19T15:49:14.293 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libradosstriper1-2:19.0.0-767.gbdd6070c.el9.x86_64 10/31 2024-01-19T15:49:14.365 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libradosstriper1-2:19.0.0-767.gbdd6070c.el9.x86_64 10/31 2024-01-19T15:49:14.409 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/31 2024-01-19T15:49:14.519 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-01-19T15:49:14.552 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-01-19T15:49:14.592 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-01-19T15:49:14.625 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-01-19T15:49:14.654 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-01-19T15:49:14.686 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-01-19T15:49:15.023 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-01-19T15:49:15.049 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librbd1-2:19.0.0-767.gbdd6070c.el9.x86_64 19/31 2024-01-19T15:49:15.149 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:19.0.0-767.gbdd6070c.el9.x86_64 19/31 2024-01-19T15:49:15.179 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rbd-2:19.0.0-767.gbdd6070c.el9.x86_64 20/31 2024-01-19T15:49:15.219 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-01-19T15:49:15.669 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-01-19T15:49:15.693 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librgw2-2:19.0.0-767.gbdd6070c.el9.x86_64 23/31 2024-01-19T15:49:15.748 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librgw2-2:19.0.0-767.gbdd6070c.el9.x86_64 23/31 2024-01-19T15:49:15.818 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rgw-2:19.0.0-767.gbdd6070c.el9.x86_64 24/31 2024-01-19T15:49:15.978 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-common-2:19.0.0-767.gbdd6070c.el9.x86 25/31 2024-01-19T15:49:18.109 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:19.0.0-767.gbdd6070c.el9.x86_64 26/31 2024-01-19T15:49:18.141 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-common-2:19.0.0-767.gbdd6070c.el9.x86_64 26/31 2024-01-19T15:49:18.776 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:19.0.0-767.gbdd6070c.el9.x86_64 26/31 2024-01-19T15:49:18.820 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-base-2:19.0.0-767.gbdd6070c.el9.x86_64 27/31 2024-01-19T15:49:18.915 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-base-2:19.0.0-767.gbdd6070c.el9.x86_64 27/31 2024-01-19T15:49:18.915 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-19T15:49:18.915 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-19T15:49:18.915 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:49:18.946 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-selinux-2:19.0.0-767.gbdd6070c.el9.x86_64 28/31 2024-01-19T15:49:28.661 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-selinux-2:19.0.0-767.gbdd6070c.el9.x86_64 28/31 2024-01-19T15:49:28.662 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /sys 2024-01-19T15:49:28.662 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /proc 2024-01-19T15:49:28.662 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /mnt 2024-01-19T15:49:28.662 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /var/tmp 2024-01-19T15:49:28.662 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /home 2024-01-19T15:49:28.662 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /root 2024-01-19T15:49:28.662 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /tmp 2024-01-19T15:49:28.662 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:49:29.520 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-radosgw-2:19.0.0-767.gbdd6070c.el9.x86_64 29/31 2024-01-19T15:49:29.544 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-767.gbdd6070c.el9.x86_64 29/31 2024-01-19T15:49:29.544 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T15:49:29.545 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-19T15:49:29.545 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-19T15:49:29.545 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-19T15:49:29.545 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:49:29.547 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-19T15:49:29.631 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-19T15:49:29.631 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-19T15:49:30.794 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-19T15:49:30.795 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-base-2:19.0.0-767.gbdd6070c.el9.x86_64 1/31 2024-01-19T15:49:30.795 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-common-2:19.0.0-767.gbdd6070c.el9.x86_64 2/31 2024-01-19T15:49:30.795 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-radosgw-2:19.0.0-767.gbdd6070c.el9.x86_64 3/31 2024-01-19T15:49:30.795 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-selinux-2:19.0.0-767.gbdd6070c.el9.x86_64 4/31 2024-01-19T15:49:30.795 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs2-2:19.0.0-767.gbdd6070c.el9.x86_64 5/31 2024-01-19T15:49:30.795 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libradosstriper1-2:19.0.0-767.gbdd6070c.el9.x86_64 6/31 2024-01-19T15:49:30.795 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librgw2-2:19.0.0-767.gbdd6070c.el9.x86_64 7/31 2024-01-19T15:49:30.795 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-argparse-2:19.0.0-767.gbdd6070c.el9.x 8/31 2024-01-19T15:49:30.795 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-common-2:19.0.0-767.gbdd6070c.el9.x86 9/31 2024-01-19T15:49:30.795 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cephfs-2:19.0.0-767.gbdd6070c.el9.x86_64 10/31 2024-01-19T15:49:30.795 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rados-2:19.0.0-767.gbdd6070c.el9.x86_64 11/31 2024-01-19T15:49:30.795 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rbd-2:19.0.0-767.gbdd6070c.el9.x86_64 12/31 2024-01-19T15:49:30.796 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rgw-2:19.0.0-767.gbdd6070c.el9.x86_64 13/31 2024-01-19T15:49:30.796 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-01-19T15:49:30.796 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-01-19T15:49:30.796 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-01-19T15:49:30.796 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-01-19T15:49:30.796 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-01-19T15:49:30.796 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-01-19T15:49:30.796 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-01-19T15:49:30.796 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-01-19T15:49:30.796 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-01-19T15:49:30.796 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-01-19T15:49:30.796 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-01-19T15:49:30.797 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-01-19T15:49:30.797 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-01-19T15:49:30.797 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-01-19T15:49:30.797 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:19.0.0-767.gbdd6070c.el9.x86_64 28/31 2024-01-19T15:49:30.797 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-01-19T15:49:30.797 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:19.0.0-767.gbdd6070c.el9.x86_64 30/31 2024-01-19T15:49:31.741 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-01-19T15:49:31.741 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:49:31.741 INFO:teuthology.orchestra.run.smithi046.stdout:Upgraded: 2024-01-19T15:49:31.741 INFO:teuthology.orchestra.run.smithi046.stdout: librados2-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:49:31.742 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:49:31.742 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-19T15:49:31.742 INFO:teuthology.orchestra.run.smithi046.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-19T15:49:31.742 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:49:31.742 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:49:31.742 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:49:31.742 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:49:31.742 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-19T15:49:31.742 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-19T15:49:31.742 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-19T15:49:31.742 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:49:31.742 INFO:teuthology.orchestra.run.smithi046.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-19T15:49:31.742 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-19T15:49:31.742 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-19T15:49:31.742 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:49:31.743 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-19T15:49:31.743 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:49:31.743 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-19T15:49:31.743 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-19T15:49:31.743 INFO:teuthology.orchestra.run.smithi046.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-19T15:49:31.743 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:49:31.743 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:49:31.743 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:49:31.743 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-19T15:49:31.743 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:49:31.743 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:49:31.743 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:49:31.743 INFO:teuthology.orchestra.run.smithi046.stdout: re2-1:20211101-3.el9.x86_64 2024-01-19T15:49:31.743 INFO:teuthology.orchestra.run.smithi046.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-19T15:49:31.744 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:49:31.744 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:49:32.066 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-test 2024-01-19T15:49:32.628 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:00:32 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:49:32.787 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:49:32.788 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:49:32.788 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-01-19T15:49:32.788 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:49:32.788 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-19T15:49:32.788 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 119 M 2024-01-19T15:49:32.788 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-01-19T15:49:32.788 INFO:teuthology.orchestra.run.smithi046.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2024-01-19T15:49:32.788 INFO:teuthology.orchestra.run.smithi046.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-19T15:49:32.788 INFO:teuthology.orchestra.run.smithi046.stdout: libasan x86_64 11.4.1-3.el9 CentOS-AppStream 417 k 2024-01-19T15:49:32.789 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 158 k 2024-01-19T15:49:32.789 INFO:teuthology.orchestra.run.smithi046.stdout: libubsan x86_64 11.4.1-3.el9 CentOS-AppStream 186 k 2024-01-19T15:49:32.789 INFO:teuthology.orchestra.run.smithi046.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-19T15:49:32.789 INFO:teuthology.orchestra.run.smithi046.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-19T15:49:32.789 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-19T15:49:32.789 INFO:teuthology.orchestra.run.smithi046.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-01-19T15:49:32.789 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:49:32.789 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-19T15:49:32.789 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:49:32.789 INFO:teuthology.orchestra.run.smithi046.stdout:Install 10 Packages 2024-01-19T15:49:32.789 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:49:32.790 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 122 M 2024-01-19T15:49:32.790 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 1.1 G 2024-01-19T15:49:32.790 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-19T15:49:33.167 INFO:teuthology.orchestra.run.smithi046.stdout:(1/10): libasan-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 417 kB 00:00 2024-01-19T15:49:33.225 INFO:teuthology.orchestra.run.smithi046.stdout:(2/10): libubsan-11.4.1-3.el9.x86_64.rpm 3.1 MB/s | 186 kB 00:00 2024-01-19T15:49:33.284 INFO:teuthology.orchestra.run.smithi046.stdout:(3/10): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-01-19T15:49:33.317 INFO:teuthology.orchestra.run.smithi046.stdout:(4/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-01-19T15:49:33.359 INFO:teuthology.orchestra.run.smithi046.stdout:(5/10): jq-1.6-16.el9.x86_64.rpm 4.4 MB/s | 186 kB 00:00 2024-01-19T15:49:33.401 INFO:teuthology.orchestra.run.smithi046.stdout:(6/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-01-19T15:49:33.543 INFO:teuthology.orchestra.run.smithi046.stdout:(7/10): libcephsqlite-19.0.0-767.gbdd6070c.el9. 248 kB/s | 158 kB 00:00 2024-01-19T15:49:33.576 INFO:teuthology.orchestra.run.smithi046.stdout:(8/10): cryptopp-8.6.0-1.el9.x86_64.rpm 7.2 MB/s | 1.3 MB 00:00 2024-01-19T15:49:33.644 INFO:teuthology.orchestra.run.smithi046.stdout:(9/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 1.2 MB/s | 122 kB 00:00 2024-01-19T15:50:17.444 INFO:teuthology.orchestra.run.smithi046.stdout:(10/10): ceph-test-19.0.0-767.gbdd6070c.el9.x86 2.7 MB/s | 119 MB 00:44 2024-01-19T15:50:17.445 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:50:17.446 INFO:teuthology.orchestra.run.smithi046.stdout:Total 2.7 MB/s | 122 MB 00:44 2024-01-19T15:50:17.446 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-19T15:50:17.470 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-19T15:50:17.470 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-19T15:50:18.330 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-19T15:50:18.330 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-19T15:50:19.518 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-19T15:50:19.602 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2024-01-19T15:50:19.640 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2024-01-19T15:50:19.680 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/10 2024-01-19T15:50:19.709 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : jq-1.6-16.el9.x86_64 4/10 2024-01-19T15:50:19.763 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/10 2024-01-19T15:50:19.791 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/10 2024-01-19T15:50:19.820 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 7/10 2024-01-19T15:50:19.862 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libasan-11.4.1-3.el9.x86_64 8/10 2024-01-19T15:50:19.889 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephsqlite-2:19.0.0-767.gbdd6070c.el9.x86_64 9/10 2024-01-19T15:50:37.316 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephsqlite-2:19.0.0-767.gbdd6070c.el9.x86_64 9/10 2024-01-19T15:50:37.397 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-test-2:19.0.0-767.gbdd6070c.el9.x86_64 10/10 2024-01-19T15:50:45.575 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-test-2:19.0.0-767.gbdd6070c.el9.x86_64 10/10 2024-01-19T15:50:45.575 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-test-2:19.0.0-767.gbdd6070c.el9.x86_64 1/10 2024-01-19T15:50:45.575 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephsqlite-2:19.0.0-767.gbdd6070c.el9.x86_64 2/10 2024-01-19T15:50:45.575 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libasan-11.4.1-3.el9.x86_64 3/10 2024-01-19T15:50:45.576 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 4/10 2024-01-19T15:50:45.576 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/10 2024-01-19T15:50:45.576 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/10 2024-01-19T15:50:45.576 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : jq-1.6-16.el9.x86_64 7/10 2024-01-19T15:50:45.576 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 8/10 2024-01-19T15:50:45.576 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2024-01-19T15:50:45.968 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2024-01-19T15:50:45.968 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:50:45.968 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-19T15:50:45.969 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test-2:19.0.0-767.gbdd6070c.el9.x86_64 cryptopp-8.6.0-1.el9.x86_64 2024-01-19T15:50:45.969 INFO:teuthology.orchestra.run.smithi046.stdout: jq-1.6-16.el9.x86_64 libasan-11.4.1-3.el9.x86_64 2024-01-19T15:50:45.969 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite-2:19.0.0-767.gbdd6070c.el9.x86_64 libubsan-11.4.1-3.el9.x86_64 2024-01-19T15:50:45.969 INFO:teuthology.orchestra.run.smithi046.stdout: oniguruma-6.9.6-1.el9.5.x86_64 socat-1.7.4.1-5.el9.x86_64 2024-01-19T15:50:45.969 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet-1.6.1-20.el9.x86_64 yaml-cpp-0.6.3-5.el9.x86_64 2024-01-19T15:50:45.969 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:50:45.969 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:50:46.384 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph 2024-01-19T15:50:46.953 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:01:46 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:50:47.116 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:50:47.119 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================== 2024-01-19T15:50:47.119 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-01-19T15:50:47.119 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================== 2024-01-19T15:50:47.119 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-19T15:50:47.119 INFO:teuthology.orchestra.run.smithi046.stdout: ceph x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 6.6 k 2024-01-19T15:50:47.119 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-01-19T15:50:47.119 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 2.0 M 2024-01-19T15:50:47.119 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 1.5 M 2024-01-19T15:50:47.119 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core noarch 2:19.0.0-767.gbdd6070c.el9 ceph-noarch 245 k 2024-01-19T15:50:47.119 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 4.8 M 2024-01-19T15:50:47.119 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 117 M 2024-01-19T15:50:47.120 INFO:teuthology.orchestra.run.smithi046.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-01-19T15:50:47.120 INFO:teuthology.orchestra.run.smithi046.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-19T15:50:47.120 INFO:teuthology.orchestra.run.smithi046.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-19T15:50:47.120 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-19T15:50:47.120 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-19T15:50:47.120 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-19T15:50:47.120 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-19T15:50:47.120 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-19T15:50:47.120 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-19T15:50:47.120 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-19T15:50:47.120 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-19T15:50:47.120 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-19T15:50:47.121 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-19T15:50:47.121 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-19T15:50:47.121 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-19T15:50:47.121 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-19T15:50:47.121 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-19T15:50:47.121 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-19T15:50:47.121 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-19T15:50:47.121 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-19T15:50:47.121 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-19T15:50:47.121 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-19T15:50:47.121 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-19T15:50:47.121 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-19T15:50:47.121 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-19T15:50:47.121 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-19T15:50:47.122 INFO:teuthology.orchestra.run.smithi046.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-19T15:50:47.122 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-19T15:50:47.122 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-19T15:50:47.122 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-19T15:50:47.122 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:50:47.122 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-19T15:50:47.122 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================== 2024-01-19T15:50:47.122 INFO:teuthology.orchestra.run.smithi046.stdout:Install 36 Packages 2024-01-19T15:50:47.122 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:50:47.123 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 130 M 2024-01-19T15:50:47.123 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 1.4 G 2024-01-19T15:50:47.123 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-19T15:50:47.529 INFO:teuthology.orchestra.run.smithi046.stdout:(1/36): ceph-19.0.0-767.gbdd6070c.el9.x86_64.rp 25 kB/s | 6.6 kB 00:00 2024-01-19T15:50:47.821 INFO:teuthology.orchestra.run.smithi046.stdout:(2/36): ceph-mgr-19.0.0-767.gbdd6070c.el9.x86_6 2.7 MB/s | 1.5 MB 00:00 2024-01-19T15:50:47.871 INFO:teuthology.orchestra.run.smithi046.stdout:(3/36): ceph-mds-19.0.0-767.gbdd6070c.el9.x86_6 3.4 MB/s | 2.0 MB 00:00 2024-01-19T15:50:47.946 INFO:teuthology.orchestra.run.smithi046.stdout:(4/36): ceph-mgr-modules-core-19.0.0-767.gbdd60 3.2 MB/s | 245 kB 00:00 2024-01-19T15:50:48.072 INFO:teuthology.orchestra.run.smithi046.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-01-19T15:50:48.113 INFO:teuthology.orchestra.run.smithi046.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-01-19T15:50:48.147 INFO:teuthology.orchestra.run.smithi046.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 953 kB/s | 32 kB 00:00 2024-01-19T15:50:48.181 INFO:teuthology.orchestra.run.smithi046.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-01-19T15:50:48.214 INFO:teuthology.orchestra.run.smithi046.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-01-19T15:50:48.256 INFO:teuthology.orchestra.run.smithi046.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-01-19T15:50:48.323 INFO:teuthology.orchestra.run.smithi046.stdout:(11/36): ceph-mon-19.0.0-767.gbdd6070c.el9.x86_ 6.1 MB/s | 4.8 MB 00:00 2024-01-19T15:50:48.373 INFO:teuthology.orchestra.run.smithi046.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 9.8 MB/s | 1.1 MB 00:00 2024-01-19T15:50:48.390 INFO:teuthology.orchestra.run.smithi046.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-01-19T15:50:48.416 INFO:teuthology.orchestra.run.smithi046.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-01-19T15:50:48.441 INFO:teuthology.orchestra.run.smithi046.stdout:(15/36): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-01-19T15:50:48.466 INFO:teuthology.orchestra.run.smithi046.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-01-19T15:50:48.516 INFO:teuthology.orchestra.run.smithi046.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 298 kB/s | 22 kB 00:00 2024-01-19T15:50:48.959 INFO:teuthology.orchestra.run.smithi046.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 98 kB/s | 43 kB 00:00 2024-01-19T15:50:49.025 INFO:teuthology.orchestra.run.smithi046.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 270 kB/s | 151 kB 00:00 2024-01-19T15:50:49.251 INFO:teuthology.orchestra.run.smithi046.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 589 kB/s | 172 kB 00:00 2024-01-19T15:50:49.276 INFO:teuthology.orchestra.run.smithi046.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-01-19T15:50:49.376 INFO:teuthology.orchestra.run.smithi046.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 85 kB/s | 11 kB 00:00 2024-01-19T15:50:49.401 INFO:teuthology.orchestra.run.smithi046.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 141 kB/s | 18 kB 00:00 2024-01-19T15:50:49.493 INFO:teuthology.orchestra.run.smithi046.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 198 kB/s | 23 kB 00:00 2024-01-19T15:50:49.519 INFO:teuthology.orchestra.run.smithi046.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 166 kB/s | 19 kB 00:00 2024-01-19T15:50:49.602 INFO:teuthology.orchestra.run.smithi046.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2024-01-19T15:50:49.678 INFO:teuthology.orchestra.run.smithi046.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 292 kB/s | 46 kB 00:00 2024-01-19T15:50:49.761 INFO:teuthology.orchestra.run.smithi046.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 497 kB/s | 79 kB 00:00 2024-01-19T15:50:49.870 INFO:teuthology.orchestra.run.smithi046.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-01-19T15:50:49.912 INFO:teuthology.orchestra.run.smithi046.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 109 kB/s | 16 kB 00:00 2024-01-19T15:50:50.020 INFO:teuthology.orchestra.run.smithi046.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 598 kB/s | 90 kB 00:00 2024-01-19T15:50:50.054 INFO:teuthology.orchestra.run.smithi046.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 252 kB/s | 36 kB 00:00 2024-01-19T15:50:50.204 INFO:teuthology.orchestra.run.smithi046.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-01-19T15:50:50.321 INFO:teuthology.orchestra.run.smithi046.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 171 kB/s | 20 kB 00:00 2024-01-19T15:50:50.355 INFO:teuthology.orchestra.run.smithi046.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.4 MB/s | 427 kB 00:00 2024-01-19T15:50:54.397 INFO:teuthology.orchestra.run.smithi046.stdout:(36/36): ceph-osd-19.0.0-767.gbdd6070c.el9.x86_ 18 MB/s | 117 MB 00:06 2024-01-19T15:50:54.402 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:50:54.402 INFO:teuthology.orchestra.run.smithi046.stdout:Total 18 MB/s | 130 MB 00:07 2024-01-19T15:50:54.417 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-19T15:50:54.464 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-19T15:50:54.464 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-19T15:50:55.495 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-19T15:50:55.496 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-19T15:50:57.429 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-19T15:50:57.490 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-01-19T15:50:57.529 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-01-19T15:50:57.561 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-01-19T15:50:57.635 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-01-19T15:50:57.685 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-01-19T15:50:57.718 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-01-19T15:50:57.814 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-01-19T15:50:57.891 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-01-19T15:50:57.938 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-01-19T15:50:58.020 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-01-19T15:50:58.072 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-01-19T15:50:58.109 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2024-01-19T15:50:58.168 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-01-19T15:50:58.232 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-01-19T15:50:58.424 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-01-19T15:50:58.480 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-01-19T15:50:58.536 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-01-19T15:50:58.578 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-01-19T15:50:58.608 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-01-19T15:50:58.688 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-01-19T15:50:58.718 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-19T15:50:58.751 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-01-19T15:50:58.792 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-01-19T15:50:58.844 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-01-19T15:50:58.878 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-01-19T15:50:58.972 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-01-19T15:50:59.009 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-01-19T15:50:59.084 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-01-19T15:50:59.228 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-01-19T15:50:59.364 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-19T15:50:59.535 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-767.gbdd6070c.el9.n 31/36 2024-01-19T15:50:59.563 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-2:19.0.0-767.gbdd6070c.el9.x86_64 32/36 2024-01-19T15:50:59.587 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-2:19.0.0-767.gbdd6070c.el9.x86_64 32/36 2024-01-19T15:50:59.587 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T15:50:59.587 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-19T15:50:59.588 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-19T15:50:59.588 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-19T15:50:59.588 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:19.095 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-osd-2:19.0.0-767.gbdd6070c.el9.x86_64 33/36 2024-01-19T15:51:19.121 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-osd-2:19.0.0-767.gbdd6070c.el9.x86_64 33/36 2024-01-19T15:51:19.121 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T15:51:19.121 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-19T15:51:19.121 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-19T15:51:19.121 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-19T15:51:19.121 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:19.604 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mon-2:19.0.0-767.gbdd6070c.el9.x86_64 34/36 2024-01-19T15:51:19.627 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mon-2:19.0.0-767.gbdd6070c.el9.x86_64 34/36 2024-01-19T15:51:19.627 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T15:51:19.627 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-19T15:51:19.627 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-19T15:51:19.627 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-19T15:51:19.627 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:19.846 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mds-2:19.0.0-767.gbdd6070c.el9.x86_64 35/36 2024-01-19T15:51:19.869 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mds-2:19.0.0-767.gbdd6070c.el9.x86_64 35/36 2024-01-19T15:51:19.869 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T15:51:19.869 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-19T15:51:19.869 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-19T15:51:19.869 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-19T15:51:19.870 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:19.947 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-2:19.0.0-767.gbdd6070c.el9.x86_64 36/36 2024-01-19T15:51:29.525 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-2:19.0.0-767.gbdd6070c.el9.x86_64 36/36 2024-01-19T15:51:29.525 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-2:19.0.0-767.gbdd6070c.el9.x86_64 1/36 2024-01-19T15:51:29.525 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mds-2:19.0.0-767.gbdd6070c.el9.x86_64 2/36 2024-01-19T15:51:29.525 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-2:19.0.0-767.gbdd6070c.el9.x86_64 3/36 2024-01-19T15:51:29.525 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mon-2:19.0.0-767.gbdd6070c.el9.x86_64 4/36 2024-01-19T15:51:29.525 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-osd-2:19.0.0-767.gbdd6070c.el9.x86_64 5/36 2024-01-19T15:51:29.525 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-767.gbdd6070c.el9.n 6/36 2024-01-19T15:51:29.525 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-01-19T15:51:29.526 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-01-19T15:51:29.526 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-01-19T15:51:29.526 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-01-19T15:51:29.526 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-01-19T15:51:29.526 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-01-19T15:51:29.526 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-01-19T15:51:29.526 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-01-19T15:51:29.526 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-01-19T15:51:29.526 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2024-01-19T15:51:29.526 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-01-19T15:51:29.526 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-01-19T15:51:29.526 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-01-19T15:51:29.526 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-01-19T15:51:29.526 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-19T15:51:29.526 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-01-19T15:51:29.527 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-01-19T15:51:29.527 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-01-19T15:51:29.527 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-01-19T15:51:29.527 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-01-19T15:51:29.527 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-01-19T15:51:29.527 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-01-19T15:51:29.527 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-01-19T15:51:29.527 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-19T15:51:29.527 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-01-19T15:51:29.527 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-01-19T15:51:29.527 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-01-19T15:51:29.527 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-01-19T15:51:29.527 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-01-19T15:51:30.359 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-01-19T15:51:30.359 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:30.359 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-19T15:51:30.359 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:51:30.359 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:51:30.359 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:51:30.360 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core-2:19.0.0-767.gbdd6070c.el9.noarch 2024-01-19T15:51:30.360 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:51:30.360 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:51:30.360 INFO:teuthology.orchestra.run.smithi046.stdout: lua-5.4.4-4.el9.x86_64 2024-01-19T15:51:30.360 INFO:teuthology.orchestra.run.smithi046.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-19T15:51:30.360 INFO:teuthology.orchestra.run.smithi046.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-19T15:51:30.360 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-19T15:51:30.360 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-19T15:51:30.360 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-19T15:51:30.360 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-19T15:51:30.360 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-19T15:51:30.360 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-19T15:51:30.360 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-19T15:51:30.361 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-19T15:51:30.361 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-19T15:51:30.361 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-19T15:51:30.361 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-19T15:51:30.361 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-19T15:51:30.361 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-19T15:51:30.361 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-19T15:51:30.361 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-19T15:51:30.361 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ply-3.11-14.el9.noarch 2024-01-19T15:51:30.361 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-19T15:51:30.361 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-19T15:51:30.361 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-19T15:51:30.361 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-19T15:51:30.361 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-19T15:51:30.361 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-19T15:51:30.362 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-19T15:51:30.362 INFO:teuthology.orchestra.run.smithi046.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-19T15:51:30.362 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-19T15:51:30.362 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-19T15:51:30.362 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-19T15:51:30.362 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:30.362 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:51:30.593 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-base 2024-01-19T15:51:31.166 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:02:31 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:51:31.269 INFO:teuthology.orchestra.run.smithi046.stdout:Package ceph-base-2:19.0.0-767.gbdd6070c.el9.x86_64 is already installed. 2024-01-19T15:51:31.322 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:51:31.323 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-01-19T15:51:31.324 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:51:31.388 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install cephadm 2024-01-19T15:51:31.958 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:02:31 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:51:32.114 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:51:32.115 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:51:32.115 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-01-19T15:51:32.115 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:51:32.115 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-19T15:51:32.115 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm noarch 2:19.0.0-767.gbdd6070c.el9 ceph-noarch 742 k 2024-01-19T15:51:32.115 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:32.115 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-19T15:51:32.115 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:51:32.115 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-01-19T15:51:32.116 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:32.116 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 742 k 2024-01-19T15:51:32.116 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 748 k 2024-01-19T15:51:32.116 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-19T15:51:32.653 INFO:teuthology.orchestra.run.smithi046.stdout:cephadm-19.0.0-767.gbdd6070c.el9.noarch.rpm 1.4 MB/s | 742 kB 00:00 2024-01-19T15:51:32.653 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:51:32.653 INFO:teuthology.orchestra.run.smithi046.stdout:Total 1.3 MB/s | 742 kB 00:00 2024-01-19T15:51:32.653 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-19T15:51:32.659 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-19T15:51:32.659 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-19T15:51:32.665 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-19T15:51:32.665 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-19T15:51:32.744 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-19T15:51:33.233 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephadm-2:19.0.0-767.gbdd6070c.el9.noarch 1/1 2024-01-19T15:51:33.326 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephadm-2:19.0.0-767.gbdd6070c.el9.noarch 1/1 2024-01-19T15:51:33.748 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephadm-2:19.0.0-767.gbdd6070c.el9.noarch 1/1 2024-01-19T15:51:34.166 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephadm-2:19.0.0-767.gbdd6070c.el9.noarch 1/1 2024-01-19T15:51:34.166 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:34.166 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-19T15:51:34.166 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm-2:19.0.0-767.gbdd6070c.el9.noarch 2024-01-19T15:51:34.166 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:34.166 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:51:34.381 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-immutable-object-cache 2024-01-19T15:51:34.955 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:02:34 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:51:35.112 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:51:35.113 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:51:35.113 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-01-19T15:51:35.113 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:51:35.113 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-19T15:51:35.113 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 147 k 2024-01-19T15:51:35.113 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:35.114 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-19T15:51:35.114 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:51:35.114 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-01-19T15:51:35.114 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:35.114 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 147 k 2024-01-19T15:51:35.114 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 481 k 2024-01-19T15:51:35.114 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-19T15:51:35.702 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-immutable-object-cache-19.0.0-767.gbdd6070 251 kB/s | 147 kB 00:00 2024-01-19T15:51:35.703 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:51:35.703 INFO:teuthology.orchestra.run.smithi046.stdout:Total 250 kB/s | 147 kB 00:00 2024-01-19T15:51:35.703 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-19T15:51:35.711 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-19T15:51:35.711 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-19T15:51:35.766 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-19T15:51:35.767 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-19T15:51:35.911 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-19T15:51:36.002 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-767.gbdd6070c.e 1/1 2024-01-19T15:51:36.062 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-767.gbdd6070c.e 1/1 2024-01-19T15:51:36.062 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T15:51:36.062 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-19T15:51:36.062 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:37.028 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-767.gbdd6070c.e 1/1 2024-01-19T15:51:37.028 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:37.028 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-19T15:51:37.028 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:51:37.028 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:37.028 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:51:37.273 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr 2024-01-19T15:51:37.846 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:02:37 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:51:37.949 INFO:teuthology.orchestra.run.smithi046.stdout:Package ceph-mgr-2:19.0.0-767.gbdd6070c.el9.x86_64 is already installed. 2024-01-19T15:51:38.003 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:51:38.004 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-01-19T15:51:38.004 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:51:38.082 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr-dashboard 2024-01-19T15:51:38.649 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:02:38 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:51:38.804 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:51:38.805 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:51:38.805 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-01-19T15:51:38.805 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:51:38.805 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-19T15:51:38.805 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard noarch 2:19.0.0-767.gbdd6070c.el9 ceph-noarch 1.7 M 2024-01-19T15:51:38.805 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-01-19T15:51:38.805 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards noarch 2:19.0.0-767.gbdd6070c.el9 ceph-noarch 28 k 2024-01-19T15:51:38.806 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts noarch 2:19.0.0-767.gbdd6070c.el9 ceph-noarch 15 k 2024-01-19T15:51:38.806 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-19T15:51:38.806 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-19T15:51:38.806 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:38.806 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-19T15:51:38.806 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:51:38.806 INFO:teuthology.orchestra.run.smithi046.stdout:Install 5 Packages 2024-01-19T15:51:38.806 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:38.807 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 2.0 M 2024-01-19T15:51:38.807 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 8.5 M 2024-01-19T15:51:38.807 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-19T15:51:39.388 INFO:teuthology.orchestra.run.smithi046.stdout:(1/5): ceph-prometheus-alerts-19.0.0-767.gbdd60 36 kB/s | 15 kB 00:00 2024-01-19T15:51:39.464 INFO:teuthology.orchestra.run.smithi046.stdout:(2/5): ceph-grafana-dashboards-19.0.0-767.gbdd6 58 kB/s | 28 kB 00:00 2024-01-19T15:51:39.489 INFO:teuthology.orchestra.run.smithi046.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 307 kB/s | 31 kB 00:00 2024-01-19T15:51:39.606 INFO:teuthology.orchestra.run.smithi046.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 1.3 MB/s | 188 kB 00:00 2024-01-19T15:51:40.856 INFO:teuthology.orchestra.run.smithi046.stdout:(5/5): ceph-mgr-dashboard-19.0.0-767.gbdd6070c. 945 kB/s | 1.7 MB 00:01 2024-01-19T15:51:40.858 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:51:40.858 INFO:teuthology.orchestra.run.smithi046.stdout:Total 996 kB/s | 2.0 MB 00:02 2024-01-19T15:51:40.859 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-19T15:51:40.867 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-19T15:51:40.867 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-19T15:51:40.940 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-19T15:51:40.941 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-19T15:51:41.187 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-19T15:51:41.280 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-01-19T15:51:41.311 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-01-19T15:51:41.341 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-767.gbdd6070c.el9.no 3/5 2024-01-19T15:51:41.588 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-767.gbdd6070c.el9.n 4/5 2024-01-19T15:51:41.626 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-767.gbdd6070c.el9.noarch 5/5 2024-01-19T15:51:42.170 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-767.gbdd6070c.el9.noarch 5/5 2024-01-19T15:51:42.170 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-767.gbdd6070c.el9.n 1/5 2024-01-19T15:51:42.170 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-767.gbdd6070c.el9.noarch 2/5 2024-01-19T15:51:42.170 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-767.gbdd6070c.el9.no 3/5 2024-01-19T15:51:42.170 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-01-19T15:51:42.466 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-01-19T15:51:42.466 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:42.467 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-19T15:51:42.467 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards-2:19.0.0-767.gbdd6070c.el9.noarch 2024-01-19T15:51:42.467 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard-2:19.0.0-767.gbdd6070c.el9.noarch 2024-01-19T15:51:42.467 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts-2:19.0.0-767.gbdd6070c.el9.noarch 2024-01-19T15:51:42.467 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-19T15:51:42.467 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-19T15:51:42.467 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:42.467 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:51:42.678 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-19T15:51:43.247 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:02:43 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:51:43.405 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:51:43.406 INFO:teuthology.orchestra.run.smithi046.stdout:=========================================================================================== 2024-01-19T15:51:43.406 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-01-19T15:51:43.406 INFO:teuthology.orchestra.run.smithi046.stdout:=========================================================================================== 2024-01-19T15:51:43.406 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-19T15:51:43.406 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-767.gbdd6070c.el9 ceph-noarch 7.4 M 2024-01-19T15:51:43.406 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-01-19T15:51:43.406 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-19T15:51:43.406 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-19T15:51:43.406 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-19T15:51:43.406 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-19T15:51:43.407 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-19T15:51:43.407 INFO:teuthology.orchestra.run.smithi046.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-19T15:51:43.407 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-19T15:51:43.407 INFO:teuthology.orchestra.run.smithi046.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-19T15:51:43.407 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-19T15:51:43.407 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-19T15:51:43.407 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-19T15:51:43.407 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:43.407 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-19T15:51:43.407 INFO:teuthology.orchestra.run.smithi046.stdout:=========================================================================================== 2024-01-19T15:51:43.407 INFO:teuthology.orchestra.run.smithi046.stdout:Install 12 Packages 2024-01-19T15:51:43.407 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:43.408 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 37 M 2024-01-19T15:51:43.408 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 209 M 2024-01-19T15:51:43.408 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-19T15:51:43.579 INFO:teuthology.orchestra.run.smithi046.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 246 kB/s | 32 kB 00:00 2024-01-19T15:51:43.612 INFO:teuthology.orchestra.run.smithi046.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 516 kB/s | 17 kB 00:00 2024-01-19T15:51:43.655 INFO:teuthology.orchestra.run.smithi046.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 851 kB/s | 34 kB 00:00 2024-01-19T15:51:43.830 INFO:teuthology.orchestra.run.smithi046.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-01-19T15:51:43.930 INFO:teuthology.orchestra.run.smithi046.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.2 MB/s | 3.0 MB 00:00 2024-01-19T15:51:44.005 INFO:teuthology.orchestra.run.smithi046.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 4.8 MB 00:00 2024-01-19T15:51:44.030 INFO:teuthology.orchestra.run.smithi046.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-01-19T15:51:44.239 INFO:teuthology.orchestra.run.smithi046.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 802 kB 00:00 2024-01-19T15:51:44.614 INFO:teuthology.orchestra.run.smithi046.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-01-19T15:51:44.640 INFO:teuthology.orchestra.run.smithi046.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 487 kB/s | 194 kB 00:00 2024-01-19T15:51:45.381 INFO:teuthology.orchestra.run.smithi046.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 331 kB/s | 245 kB 00:00 2024-01-19T15:51:47.840 INFO:teuthology.orchestra.run.smithi046.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-7 1.7 MB/s | 7.4 MB 00:04 2024-01-19T15:51:47.844 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:51:47.844 INFO:teuthology.orchestra.run.smithi046.stdout:Total 8.4 MB/s | 37 MB 00:04 2024-01-19T15:51:47.858 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-19T15:51:47.886 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-19T15:51:47.887 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-19T15:51:48.304 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-19T15:51:48.305 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-19T15:51:49.145 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-19T15:51:49.235 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-01-19T15:51:49.269 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-01-19T15:51:49.380 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-01-19T15:51:49.423 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-01-19T15:51:49.612 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-19T15:51:49.638 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-19T15:51:49.718 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-01-19T15:51:50.405 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-01-19T15:51:50.523 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-01-19T15:51:51.825 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-01-19T15:51:52.902 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-01-19T15:51:52.940 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-767.gbdd607 12/12 2024-01-19T15:51:55.417 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-767.gbdd607 12/12 2024-01-19T15:51:55.418 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-767.gbdd607 1/12 2024-01-19T15:51:55.418 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-01-19T15:51:55.418 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-01-19T15:51:55.418 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-01-19T15:51:55.418 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-19T15:51:55.419 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-19T15:51:55.419 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-01-19T15:51:55.419 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-01-19T15:51:55.419 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-01-19T15:51:55.419 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-01-19T15:51:55.419 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-01-19T15:51:55.859 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-01-19T15:51:55.859 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:55.859 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-19T15:51:55.859 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local-2:19.0.0-767.gbdd6070c.el9.noarch 2024-01-19T15:51:55.859 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-19T15:51:55.860 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-19T15:51:55.860 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-19T15:51:55.860 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-19T15:51:55.860 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-19T15:51:55.860 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-19T15:51:55.860 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-19T15:51:55.860 INFO:teuthology.orchestra.run.smithi046.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-19T15:51:55.860 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-19T15:51:55.860 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-19T15:51:55.860 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-19T15:51:55.860 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:55.860 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:51:56.245 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr-rook 2024-01-19T15:51:56.814 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:02:56 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:51:56.975 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:51:56.976 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================= 2024-01-19T15:51:56.976 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-01-19T15:51:56.976 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================= 2024-01-19T15:51:56.976 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-19T15:51:56.976 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook noarch 2:19.0.0-767.gbdd6070c.el9 ceph-noarch 51 k 2024-01-19T15:51:56.977 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-01-19T15:51:56.977 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-19T15:51:56.977 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-19T15:51:56.977 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth noarch 1:2.26.0-1.el9 epel 202 k 2024-01-19T15:51:56.977 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-19T15:51:56.977 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-19T15:51:56.978 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-19T15:51:56.978 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-19T15:51:56.978 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-19T15:51:56.978 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-19T15:51:56.978 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-19T15:51:56.978 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-19T15:51:56.979 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-19T15:51:56.979 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:56.979 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-19T15:51:56.979 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================= 2024-01-19T15:51:56.979 INFO:teuthology.orchestra.run.smithi046.stdout:Install 13 Packages 2024-01-19T15:51:56.979 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:51:56.980 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 2.0 M 2024-01-19T15:51:56.981 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 26 M 2024-01-19T15:51:56.981 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-19T15:51:57.473 INFO:teuthology.orchestra.run.smithi046.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 173 kB/s | 24 kB 00:00 2024-01-19T15:51:57.499 INFO:teuthology.orchestra.run.smithi046.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 109 kB/s | 18 kB 00:00 2024-01-19T15:51:57.566 INFO:teuthology.orchestra.run.smithi046.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 174 kB 00:00 2024-01-19T15:51:57.591 INFO:teuthology.orchestra.run.smithi046.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-01-19T15:51:57.616 INFO:teuthology.orchestra.run.smithi046.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-01-19T15:51:57.633 INFO:teuthology.orchestra.run.smithi046.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-01-19T15:51:57.708 INFO:teuthology.orchestra.run.smithi046.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 199 kB/s | 15 kB 00:00 2024-01-19T15:51:57.742 INFO:teuthology.orchestra.run.smithi046.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 256 kB/s | 32 kB 00:00 2024-01-19T15:51:57.818 INFO:teuthology.orchestra.run.smithi046.stdout:(9/13): ceph-mgr-rook-19.0.0-767.gbdd6070c.el9. 105 kB/s | 51 kB 00:00 2024-01-19T15:51:57.859 INFO:teuthology.orchestra.run.smithi046.stdout:(10/13): python3-kubernetes-26.1.0-2.el9.noarch 8.7 MB/s | 1.0 MB 00:00 2024-01-19T15:51:57.884 INFO:teuthology.orchestra.run.smithi046.stdout:(11/13): python3-google-auth-2.26.0-1.el9.noarc 1.1 MB/s | 202 kB 00:00 2024-01-19T15:51:57.926 INFO:teuthology.orchestra.run.smithi046.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 543 kB/s | 59 kB 00:00 2024-01-19T15:51:57.952 INFO:teuthology.orchestra.run.smithi046.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 973 kB/s | 90 kB 00:00 2024-01-19T15:51:57.953 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:51:57.954 INFO:teuthology.orchestra.run.smithi046.stdout:Total 2.1 MB/s | 2.0 MB 00:00 2024-01-19T15:51:57.954 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-19T15:51:57.968 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-19T15:51:57.968 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-19T15:51:58.070 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-19T15:51:58.070 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-19T15:51:58.623 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-19T15:51:58.798 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-01-19T15:51:58.855 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-01-19T15:51:58.919 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-01-19T15:51:58.954 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-01-19T15:51:58.996 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-01-19T15:51:59.093 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-01-19T15:51:59.213 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-google-auth-1:2.26.0-1.el9.noarch 7/13 2024-01-19T15:51:59.265 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-01-19T15:52:00.043 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-01-19T15:52:00.094 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-01-19T15:52:00.128 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-01-19T15:52:00.176 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-01-19T15:52:00.207 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-rook-2:19.0.0-767.gbdd6070c.el9.noarch 13/13 2024-01-19T15:52:01.030 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-767.gbdd6070c.el9.noarch 13/13 2024-01-19T15:52:01.031 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-rook-2:19.0.0-767.gbdd6070c.el9.noarch 1/13 2024-01-19T15:52:01.031 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-01-19T15:52:01.031 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-01-19T15:52:01.031 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-01-19T15:52:01.031 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-01-19T15:52:01.031 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-01-19T15:52:01.031 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-01-19T15:52:01.031 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-01-19T15:52:01.031 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-01-19T15:52:01.032 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-google-auth-1:2.26.0-1.el9.noarch 10/13 2024-01-19T15:52:01.032 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-01-19T15:52:01.032 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-01-19T15:52:01.419 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-01-19T15:52:01.419 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:01.419 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-19T15:52:01.419 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook-2:19.0.0-767.gbdd6070c.el9.noarch 2024-01-19T15:52:01.419 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-19T15:52:01.419 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-19T15:52:01.419 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth-1:2.26.0-1.el9.noarch 2024-01-19T15:52:01.419 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-19T15:52:01.419 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-19T15:52:01.419 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-19T15:52:01.420 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-19T15:52:01.420 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-19T15:52:01.420 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-19T15:52:01.420 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-19T15:52:01.420 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-19T15:52:01.420 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-19T15:52:01.420 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:01.420 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:52:01.688 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-mgr-cephadm 2024-01-19T15:52:02.256 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:03:02 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:52:02.415 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:52:02.416 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================= 2024-01-19T15:52:02.416 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-01-19T15:52:02.416 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================= 2024-01-19T15:52:02.416 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-19T15:52:02.416 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm noarch 2:19.0.0-767.gbdd6070c.el9 ceph-noarch 133 k 2024-01-19T15:52:02.416 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-01-19T15:52:02.417 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-19T15:52:02.417 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-19T15:52:02.417 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-19T15:52:02.417 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-19T15:52:02.417 INFO:teuthology.orchestra.run.smithi046.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-19T15:52:02.417 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:02.417 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-19T15:52:02.417 INFO:teuthology.orchestra.run.smithi046.stdout:======================================================================================= 2024-01-19T15:52:02.417 INFO:teuthology.orchestra.run.smithi046.stdout:Install 6 Packages 2024-01-19T15:52:02.417 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:02.418 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 6.8 M 2024-01-19T15:52:02.418 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 33 M 2024-01-19T15:52:02.418 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-19T15:52:02.746 INFO:teuthology.orchestra.run.smithi046.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2024-01-19T15:52:02.829 INFO:teuthology.orchestra.run.smithi046.stdout:(2/6): ceph-mgr-cephadm-19.0.0-767.gbdd6070c.el 522 kB/s | 133 kB 00:00 2024-01-19T15:52:02.905 INFO:teuthology.orchestra.run.smithi046.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-01-19T15:52:03.055 INFO:teuthology.orchestra.run.smithi046.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 259 kB/s | 58 kB 00:00 2024-01-19T15:52:03.088 INFO:teuthology.orchestra.run.smithi046.stdout:(5/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 1.6 MB/s | 548 kB 00:00 2024-01-19T15:52:03.114 INFO:teuthology.orchestra.run.smithi046.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 244 kB/s | 51 kB 00:00 2024-01-19T15:52:03.115 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:52:03.116 INFO:teuthology.orchestra.run.smithi046.stdout:Total 9.7 MB/s | 6.8 MB 00:00 2024-01-19T15:52:03.116 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-19T15:52:03.125 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-19T15:52:03.125 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-19T15:52:03.217 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-19T15:52:03.217 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-19T15:52:03.526 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-19T15:52:03.711 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-01-19T15:52:03.775 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-01-19T15:52:04.335 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-01-19T15:52:04.413 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-01-19T15:52:04.480 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-01-19T15:52:04.504 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-767.gbdd6070c.el9.noarch 6/6 2024-01-19T15:52:05.249 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-767.gbdd6070c.el9.noarch 6/6 2024-01-19T15:52:05.249 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-767.gbdd6070c.el9.noarch 1/6 2024-01-19T15:52:05.249 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-01-19T15:52:05.249 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-01-19T15:52:05.250 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-01-19T15:52:05.250 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-01-19T15:52:05.680 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-01-19T15:52:05.681 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:05.681 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-19T15:52:05.681 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm-2:19.0.0-767.gbdd6070c.el9.noarch 2024-01-19T15:52:05.681 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-19T15:52:05.681 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-19T15:52:05.681 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-19T15:52:05.681 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-19T15:52:05.681 INFO:teuthology.orchestra.run.smithi046.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-19T15:52:05.681 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:05.681 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:52:05.940 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-fuse 2024-01-19T15:52:06.507 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:03:06 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:52:06.667 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:52:06.667 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:52:06.668 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-01-19T15:52:06.668 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:52:06.668 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-19T15:52:06.668 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 858 k 2024-01-19T15:52:06.668 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-01-19T15:52:06.668 INFO:teuthology.orchestra.run.smithi046.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-19T15:52:06.668 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:06.668 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-19T15:52:06.668 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:52:06.668 INFO:teuthology.orchestra.run.smithi046.stdout:Install 2 Packages 2024-01-19T15:52:06.668 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:06.669 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 937 k 2024-01-19T15:52:06.669 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 3.1 M 2024-01-19T15:52:06.669 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-19T15:52:06.807 INFO:teuthology.orchestra.run.smithi046.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 570 kB/s | 79 kB 00:00 2024-01-19T15:52:06.999 INFO:teuthology.orchestra.run.smithi046.stdout:(2/2): ceph-fuse-19.0.0-767.gbdd6070c.el9.x86_6 2.5 MB/s | 858 kB 00:00 2024-01-19T15:52:07.000 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:52:07.000 INFO:teuthology.orchestra.run.smithi046.stdout:Total 2.8 MB/s | 937 kB 00:00 2024-01-19T15:52:07.000 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-19T15:52:07.022 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-19T15:52:07.022 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-19T15:52:07.100 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-19T15:52:07.100 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-19T15:52:07.292 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-19T15:52:07.431 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-01-19T15:52:07.502 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-fuse-2:19.0.0-767.gbdd6070c.el9.x86_64 2/2 2024-01-19T15:52:08.199 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-fuse-2:19.0.0-767.gbdd6070c.el9.x86_64 2/2 2024-01-19T15:52:08.200 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-fuse-2:19.0.0-767.gbdd6070c.el9.x86_64 1/2 2024-01-19T15:52:08.514 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-01-19T15:52:08.514 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:08.514 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-19T15:52:08.514 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse-2:19.0.0-767.gbdd6070c.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-01-19T15:52:08.514 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:08.514 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:52:08.919 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-volume 2024-01-19T15:52:09.490 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:03:09 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:52:09.647 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:52:09.648 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:52:09.648 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-01-19T15:52:09.648 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:52:09.648 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-19T15:52:09.649 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume noarch 2:19.0.0-767.gbdd6070c.el9 ceph-noarch 262 k 2024-01-19T15:52:09.649 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:09.649 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-19T15:52:09.649 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:52:09.649 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-01-19T15:52:09.649 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:09.649 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 262 k 2024-01-19T15:52:09.649 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 1.2 M 2024-01-19T15:52:09.649 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-19T15:52:09.896 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-volume-19.0.0-767.gbdd6070c.el9.noarch.rpm 1.0 MB/s | 262 kB 00:00 2024-01-19T15:52:09.896 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:52:09.896 INFO:teuthology.orchestra.run.smithi046.stdout:Total 1.0 MB/s | 262 kB 00:00 2024-01-19T15:52:09.896 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-19T15:52:09.905 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-19T15:52:09.905 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-19T15:52:09.944 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-19T15:52:09.944 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-19T15:52:10.174 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-19T15:52:10.247 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-volume-2:19.0.0-767.gbdd6070c.el9.noarch 1/1 2024-01-19T15:52:10.263 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-volume-2:19.0.0-767.gbdd6070c.el9.noarch 1/1 2024-01-19T15:52:10.263 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T15:52:10.263 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-19T15:52:10.263 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:11.329 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-volume-2:19.0.0-767.gbdd6070c.el9.noarch 1/1 2024-01-19T15:52:11.329 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:11.329 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-19T15:52:11.329 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume-2:19.0.0-767.gbdd6070c.el9.noarch 2024-01-19T15:52:11.329 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:11.329 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:52:11.610 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install librados-devel 2024-01-19T15:52:12.182 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:03:12 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:52:12.342 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:52:12.342 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:52:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-01-19T15:52:12.343 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:52:12.343 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-19T15:52:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 127 k 2024-01-19T15:52:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:12.343 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-19T15:52:12.343 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:52:12.343 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-01-19T15:52:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:12.343 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 127 k 2024-01-19T15:52:12.343 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 457 k 2024-01-19T15:52:12.343 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-19T15:52:12.568 INFO:teuthology.orchestra.run.smithi046.stdout:librados-devel-19.0.0-767.gbdd6070c.el9.x86_64. 565 kB/s | 127 kB 00:00 2024-01-19T15:52:12.568 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:52:12.568 INFO:teuthology.orchestra.run.smithi046.stdout:Total 562 kB/s | 127 kB 00:00 2024-01-19T15:52:12.569 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-19T15:52:12.574 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-19T15:52:12.574 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-19T15:52:12.631 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-19T15:52:12.631 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-19T15:52:12.778 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-19T15:52:12.886 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librados-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:52:13.365 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:52:13.685 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:52:13.686 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:13.686 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-19T15:52:13.686 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:52:13.686 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:13.686 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:52:13.936 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install libcephfs2 2024-01-19T15:52:14.515 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:03:14 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:52:14.619 INFO:teuthology.orchestra.run.smithi046.stdout:Package libcephfs2-2:19.0.0-767.gbdd6070c.el9.x86_64 is already installed. 2024-01-19T15:52:14.674 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:52:14.675 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-01-19T15:52:14.675 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:52:14.754 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install libcephfs-devel 2024-01-19T15:52:15.322 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:03:15 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:52:15.485 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:52:15.485 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:52:15.485 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-01-19T15:52:15.485 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:52:15.485 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-19T15:52:15.486 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 31 k 2024-01-19T15:52:15.486 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:15.486 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-19T15:52:15.486 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:52:15.486 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-01-19T15:52:15.486 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:15.486 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 31 k 2024-01-19T15:52:15.486 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 140 k 2024-01-19T15:52:15.486 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-19T15:52:15.730 INFO:teuthology.orchestra.run.smithi046.stdout:libcephfs-devel-19.0.0-767.gbdd6070c.el9.x86_64 128 kB/s | 31 kB 00:00 2024-01-19T15:52:15.730 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:52:15.731 INFO:teuthology.orchestra.run.smithi046.stdout:Total 127 kB/s | 31 kB 00:00 2024-01-19T15:52:15.731 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-19T15:52:15.734 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-19T15:52:15.734 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-19T15:52:15.755 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-19T15:52:15.755 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-19T15:52:15.850 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-19T15:52:15.971 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:52:16.353 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:52:16.620 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:52:16.621 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:16.621 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-19T15:52:16.621 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:52:16.621 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:16.621 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:52:16.823 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install librados2 2024-01-19T15:52:17.388 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:03:17 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:52:17.491 INFO:teuthology.orchestra.run.smithi046.stdout:Package librados2-2:19.0.0-767.gbdd6070c.el9.x86_64 is already installed. 2024-01-19T15:52:17.545 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:52:17.546 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-01-19T15:52:17.547 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:52:17.630 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install librbd1 2024-01-19T15:52:18.197 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:03:18 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:52:18.301 INFO:teuthology.orchestra.run.smithi046.stdout:Package librbd1-2:19.0.0-767.gbdd6070c.el9.x86_64 is already installed. 2024-01-19T15:52:18.355 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:52:18.356 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-01-19T15:52:18.356 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:52:18.436 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-rados 2024-01-19T15:52:19.025 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:03:19 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:52:19.130 INFO:teuthology.orchestra.run.smithi046.stdout:Package python3-rados-2:19.0.0-767.gbdd6070c.el9.x86_64 is already installed. 2024-01-19T15:52:19.184 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:52:19.185 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-01-19T15:52:19.185 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:52:19.271 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-rgw 2024-01-19T15:52:19.843 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:03:19 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:52:19.946 INFO:teuthology.orchestra.run.smithi046.stdout:Package python3-rgw-2:19.0.0-767.gbdd6070c.el9.x86_64 is already installed. 2024-01-19T15:52:20.000 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:52:20.001 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-01-19T15:52:20.001 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:52:20.077 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-cephfs 2024-01-19T15:52:20.646 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:03:20 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:52:20.750 INFO:teuthology.orchestra.run.smithi046.stdout:Package python3-cephfs-2:19.0.0-767.gbdd6070c.el9.x86_64 is already installed. 2024-01-19T15:52:20.804 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:52:20.805 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-01-19T15:52:20.805 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:52:20.888 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-rbd 2024-01-19T15:52:21.459 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:03:21 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:52:21.563 INFO:teuthology.orchestra.run.smithi046.stdout:Package python3-rbd-2:19.0.0-767.gbdd6070c.el9.x86_64 is already installed. 2024-01-19T15:52:21.617 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:52:21.618 INFO:teuthology.orchestra.run.smithi046.stdout:Nothing to do. 2024-01-19T15:52:21.618 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:52:21.692 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install rbd-fuse 2024-01-19T15:52:22.259 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:03:22 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:52:22.418 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:52:22.419 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:52:22.419 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-01-19T15:52:22.419 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:52:22.419 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-19T15:52:22.419 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 86 k 2024-01-19T15:52:22.419 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:22.419 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-19T15:52:22.419 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:52:22.419 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-01-19T15:52:22.420 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:22.420 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 86 k 2024-01-19T15:52:22.420 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 256 k 2024-01-19T15:52:22.420 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-19T15:52:22.825 INFO:teuthology.orchestra.run.smithi046.stdout:rbd-fuse-19.0.0-767.gbdd6070c.el9.x86_64.rpm 212 kB/s | 86 kB 00:00 2024-01-19T15:52:22.825 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:52:22.825 INFO:teuthology.orchestra.run.smithi046.stdout:Total 211 kB/s | 86 kB 00:00 2024-01-19T15:52:22.826 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-19T15:52:22.833 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-19T15:52:22.833 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-19T15:52:22.891 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-19T15:52:22.891 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-19T15:52:23.038 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-19T15:52:23.142 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-fuse-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:52:23.550 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-fuse-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:52:23.892 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-fuse-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:52:23.893 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:23.893 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-19T15:52:23.893 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:52:23.893 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:23.893 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:52:24.154 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install rbd-mirror 2024-01-19T15:52:24.739 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:03:24 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:52:24.901 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:52:24.902 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:52:24.902 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repo Size 2024-01-19T15:52:24.903 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:52:24.903 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-19T15:52:24.903 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 3.0 M 2024-01-19T15:52:24.903 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:24.903 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-19T15:52:24.903 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:52:24.903 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-01-19T15:52:24.903 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:24.903 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 3.0 M 2024-01-19T15:52:24.903 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 14 M 2024-01-19T15:52:24.903 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-19T15:52:25.704 INFO:teuthology.orchestra.run.smithi046.stdout:rbd-mirror-19.0.0-767.gbdd6070c.el9.x86_64.rpm 3.8 MB/s | 3.0 MB 00:00 2024-01-19T15:52:25.704 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:52:25.705 INFO:teuthology.orchestra.run.smithi046.stdout:Total 3.8 MB/s | 3.0 MB 00:00 2024-01-19T15:52:25.705 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-19T15:52:25.715 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-19T15:52:25.716 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-19T15:52:25.792 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-19T15:52:25.793 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-19T15:52:26.246 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-19T15:52:26.313 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-mirror-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:52:26.338 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-mirror-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:52:26.338 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T15:52:26.338 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-19T15:52:26.338 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-19T15:52:26.338 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-19T15:52:26.338 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:27.552 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-mirror-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:52:27.552 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:27.552 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-19T15:52:27.552 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:52:27.552 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:27.553 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:52:27.801 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install rbd-nbd 2024-01-19T15:52:28.376 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:03:28 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:52:28.534 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:52:28.535 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:52:28.535 INFO:teuthology.orchestra.run.smithi046.stdout: Package Architecture Version Repository Size 2024-01-19T15:52:28.535 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:52:28.535 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-19T15:52:28.535 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 169 k 2024-01-19T15:52:28.535 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:28.535 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-19T15:52:28.535 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:52:28.535 INFO:teuthology.orchestra.run.smithi046.stdout:Install 1 Package 2024-01-19T15:52:28.535 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:28.536 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 169 k 2024-01-19T15:52:28.536 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 537 k 2024-01-19T15:52:28.536 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-19T15:52:28.882 INFO:teuthology.orchestra.run.smithi046.stdout:rbd-nbd-19.0.0-767.gbdd6070c.el9.x86_64.rpm 489 kB/s | 169 kB 00:00 2024-01-19T15:52:28.883 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:52:28.883 INFO:teuthology.orchestra.run.smithi046.stdout:Total 487 kB/s | 169 kB 00:00 2024-01-19T15:52:28.883 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-19T15:52:28.891 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-19T15:52:28.891 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-19T15:52:28.949 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-19T15:52:28.949 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-19T15:52:29.106 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-19T15:52:29.209 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-nbd-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:52:29.637 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-nbd-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:52:29.936 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-nbd-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:52:29.936 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:29.936 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-19T15:52:29.936 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:52:29.936 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:29.936 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:52:30.212 DEBUG:teuthology.parallel:result is None 2024-01-19T15:52:30.213 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4 2024-01-19T15:52:30.357 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-19T15:52:30.377 INFO:teuthology.orchestra.run.smithi046.stdout:19.0.0-767.gbdd6070c.el9 2024-01-19T15:52:30.378 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-767.gbdd6070c.el9 2024-01-19T15:52:30.378 INFO:teuthology.task.install:The correct ceph version 19.0.0-767.gbdd6070c is installed. 2024-01-19T15:52:30.380 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-19T15:52:30.380 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-19T15:52:30.380 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-19T15:52:30.445 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-19T15:52:30.445 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-19T15:52:30.445 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/daemon-helper 2024-01-19T15:52:30.511 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-19T15:52:30.578 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-19T15:52:30.579 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-19T15:52:30.579 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-19T15:52:30.644 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-19T15:52:30.711 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-19T15:52:30.712 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-19T15:52:30.712 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/stdin-killer 2024-01-19T15:52:30.778 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-19T15:52:30.844 INFO:teuthology.run_tasks:Running task ceph... 2024-01-19T15:52:30.929 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-19T15:52:30.929 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 777 /var/log/ceph 2024-01-19T15:52:30.955 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-19T15:52:30.956 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-19T15:52:31.022 INFO:tasks.ceph:Creating extra log directories... 2024-01-19T15:52:31.022 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-19T15:52:31.097 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-19T15:52:31.097 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson osd obc lru size': 10, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-19T15:52:31.097 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yingxin-2024-01-19_07:32:18-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-8-distro-default-smithi/7521657', 'branch': 'wip-yingxin-crimson-save-conn-foreign-copy-8', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read}', 'email': 'yingxin.cheng@intel.com', 'first_in_suite': False, 'job_id': '7521657', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yingxin-2024-01-19_07:32:18-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-8-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yingxin-crimson-save-conn-foreign-copy-8'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson osd obc lru size': 10, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': 'bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4'}}, 'workunit': {'branch': 'wip-yingxin-crimson-save-conn-foreign-copy-8', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4'}}, 'owner': 'scheduled_yingxin@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'sha1': 'bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-yingxin-crimson-save-conn-foreign-copy-8', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4', 'targets': {'smithi046.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIEuHVGCLxJS4bLidw0RCCIWGTwemGpysryduvv9ZtUC9CsMI8rzNHbsqgtTiTlXeWgF/Tr87RV8+hsaqzDi3Mw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson osd obc lru size': 10, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randread'], 'norandommap': True, 'op_size': [4096], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2]}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'cd45576300487d997e5a85abed65500b9f5d143b', 'timestamp': '2024-01-19_07:32:18', 'tube': 'smithi', 'user': 'yingxin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-19T15:52:31.098 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-19T15:52:31.144 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m0777 -- /var/run/ceph 2024-01-19T15:52:31.211 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-19T15:52:31.211 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/scratch_devs of=/dev/stdout 2024-01-19T15:52:31.266 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-19T15:52:31.266 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_1 2024-01-19T15:52:31.320 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-19T15:52:31.320 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-19T15:52:31.320 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 800 Links: 1 2024-01-19T15:52:31.320 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T15:52:31.321 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-01-19T15:52:31.321 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-01-19 15:52:26.600776061 +0000 2024-01-19T15:52:31.321 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-01-19 15:48:10.563158057 +0000 2024-01-19T15:52:31.321 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-01-19 15:48:10.563158057 +0000 2024-01-19T15:52:31.321 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-01-19 15:48:10.563158057 +0000 2024-01-19T15:52:31.321 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-19T15:52:31.385 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-01-19T15:52:31.385 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-01-19T15:52:31.385 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 7.7656e-05 s, 6.6 MB/s 2024-01-19T15:52:31.386 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-19T15:52:31.441 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_2 2024-01-19T15:52:31.495 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-19T15:52:31.495 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-19T15:52:31.495 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 818 Links: 1 2024-01-19T15:52:31.495 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T15:52:31.495 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-01-19T15:52:31.496 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-01-19 15:52:26.601776051 +0000 2024-01-19T15:52:31.496 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-01-19 15:48:10.895154541 +0000 2024-01-19T15:52:31.496 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-01-19 15:48:10.895154541 +0000 2024-01-19T15:52:31.496 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-01-19 15:48:10.895154541 +0000 2024-01-19T15:52:31.496 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-19T15:52:31.559 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-01-19T15:52:31.560 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-01-19T15:52:31.560 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 9.3097e-05 s, 5.5 MB/s 2024-01-19T15:52:31.561 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-19T15:52:31.616 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_3 2024-01-19T15:52:31.670 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-19T15:52:31.670 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-19T15:52:31.670 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 838 Links: 1 2024-01-19T15:52:31.670 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T15:52:31.670 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-01-19T15:52:31.671 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-01-19 15:52:26.601776051 +0000 2024-01-19T15:52:31.671 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-01-19 15:48:11.223151068 +0000 2024-01-19T15:52:31.671 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-01-19 15:48:11.223151068 +0000 2024-01-19T15:52:31.671 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-01-19 15:48:11.223151068 +0000 2024-01-19T15:52:31.671 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-19T15:52:31.734 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-01-19T15:52:31.735 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-01-19T15:52:31.735 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 8.9795e-05 s, 5.7 MB/s 2024-01-19T15:52:31.736 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-19T15:52:31.791 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_4 2024-01-19T15:52:31.845 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-19T15:52:31.845 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-19T15:52:31.846 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 857 Links: 1 2024-01-19T15:52:31.846 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T15:52:31.846 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-01-19T15:52:31.846 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-01-19 15:52:26.601776051 +0000 2024-01-19T15:52:31.846 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-01-19 15:48:11.562147480 +0000 2024-01-19T15:52:31.846 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-01-19 15:48:11.562147480 +0000 2024-01-19T15:52:31.846 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-01-19 15:48:11.562147480 +0000 2024-01-19T15:52:31.846 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-19T15:52:31.909 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-01-19T15:52:31.910 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-01-19T15:52:31.910 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 9.4704e-05 s, 5.4 MB/s 2024-01-19T15:52:31.911 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-19T15:52:31.966 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-19T15:52:31.966 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi046.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-01-19T15:52:31.966 INFO:tasks.ceph:Generating config... 2024-01-19T15:52:31.967 INFO:tasks.ceph:[osd] debug monc = 20 2024-01-19T15:52:31.968 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-01-19T15:52:31.968 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-01-19T15:52:31.968 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-01-19T15:52:31.968 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-01-19T15:52:31.968 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-01-19T15:52:31.968 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-01-19T15:52:31.968 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2024-01-19T15:52:31.968 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-01-19T15:52:31.968 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-01-19T15:52:31.968 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-01-19T15:52:31.968 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-01-19T15:52:31.968 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-01-19T15:52:31.968 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-01-19T15:52:31.968 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-01-19T15:52:31.968 INFO:tasks.ceph:[osd] err to stderr = True 2024-01-19T15:52:31.969 INFO:tasks.ceph:[osd] log flush on exit = True 2024-01-19T15:52:31.969 INFO:tasks.ceph:[osd] log to file = False 2024-01-19T15:52:31.969 INFO:tasks.ceph:[osd] log to stderr = True 2024-01-19T15:52:31.969 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-01-19T15:52:31.969 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-01-19T15:52:31.969 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-01-19T15:52:31.969 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-01-19T15:52:31.969 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-01-19T15:52:31.969 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-01-19T15:52:31.969 INFO:tasks.ceph:[global] auth client required = none 2024-01-19T15:52:31.969 INFO:tasks.ceph:[global] auth cluster required = none 2024-01-19T15:52:31.969 INFO:tasks.ceph:[global] auth service required = none 2024-01-19T15:52:31.969 INFO:tasks.ceph:[global] auth supported = none 2024-01-19T15:52:31.969 INFO:tasks.ceph:[global] debug asok = 0/0 2024-01-19T15:52:31.969 INFO:tasks.ceph:[global] debug auth = 0/0 2024-01-19T15:52:31.970 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-01-19T15:52:31.970 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-01-19T15:52:31.970 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-01-19T15:52:31.970 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-01-19T15:52:31.970 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-01-19T15:52:31.970 INFO:tasks.ceph:[global] debug client = 0/0 2024-01-19T15:52:31.970 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-01-19T15:52:31.970 INFO:tasks.ceph:[global] debug context = 0/0 2024-01-19T15:52:31.970 INFO:tasks.ceph:[global] debug crush = 0/0 2024-01-19T15:52:31.970 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-01-19T15:52:31.970 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-01-19T15:52:31.970 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-01-19T15:52:31.970 INFO:tasks.ceph:[global] debug filer = 0/0 2024-01-19T15:52:31.970 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-01-19T15:52:31.970 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-01-19T15:52:31.970 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-01-19T15:52:31.971 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-01-19T15:52:31.971 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-01-19T15:52:31.971 INFO:tasks.ceph:[global] debug journal = 0/0 2024-01-19T15:52:31.971 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-01-19T15:52:31.971 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-01-19T15:52:31.971 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-01-19T15:52:31.971 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-01-19T15:52:31.971 INFO:tasks.ceph:[global] debug mds = 0/0 2024-01-19T15:52:31.971 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-01-19T15:52:31.971 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-01-19T15:52:31.971 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-01-19T15:52:31.971 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-01-19T15:52:31.971 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-01-19T15:52:31.971 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-01-19T15:52:31.971 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-01-19T15:52:31.971 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-01-19T15:52:31.971 INFO:tasks.ceph:[global] debug mon = 0/0 2024-01-19T15:52:31.972 INFO:tasks.ceph:[global] debug monc = 0/0 2024-01-19T15:52:31.972 INFO:tasks.ceph:[global] debug ms = 0/0 2024-01-19T15:52:31.972 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-01-19T15:52:31.972 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-01-19T15:52:31.972 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-01-19T15:52:31.972 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-01-19T15:52:31.972 INFO:tasks.ceph:[global] debug osd = 0/0 2024-01-19T15:52:31.972 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-01-19T15:52:31.972 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-01-19T15:52:31.972 INFO:tasks.ceph:[global] debug rados = 0/0 2024-01-19T15:52:31.972 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-01-19T15:52:31.972 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-01-19T15:52:31.972 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-01-19T15:52:31.972 INFO:tasks.ceph:[global] debug refs = 0/0 2024-01-19T15:52:31.972 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-01-19T15:52:31.973 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-01-19T15:52:31.973 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-01-19T15:52:31.973 INFO:tasks.ceph:[global] debug striper = 0/0 2024-01-19T15:52:31.973 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-01-19T15:52:31.973 INFO:tasks.ceph:[global] debug timer = 0/0 2024-01-19T15:52:31.973 INFO:tasks.ceph:[global] debug tp = 0/0 2024-01-19T15:52:31.973 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-01-19T15:52:31.973 INFO:tasks.ceph:[global] ms client mode = crc 2024-01-19T15:52:31.973 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-01-19T15:52:31.973 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-01-19T15:52:31.973 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-01-19T15:52:31.973 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-01-19T15:52:31.973 INFO:tasks.ceph:[global] ms service mode = crc 2024-01-19T15:52:31.973 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-19T15:52:31.973 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-19T15:52:31.973 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-01-19T15:52:31.974 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-01-19T15:52:31.974 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-01-19T15:52:31.974 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-01-19T15:52:31.974 INFO:tasks.ceph:Setting up mon.a... 2024-01-19T15:52:31.974 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-19T15:52:32.068 INFO:teuthology.orchestra.run.smithi046.stdout:creating /etc/ceph/ceph.keyring 2024-01-19T15:52:32.071 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-19T15:52:32.120 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-19T15:52:32.146 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.46')] 2024-01-19T15:52:32.146 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.46', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson osd obc lru size': 10, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2024-01-19T15:52:32.147 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-19T15:52:32.147 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-19T15:52:32.200 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.46 --print /home/ubuntu/cephtest/ceph.monmap 2024-01-19T15:52:32.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:52:32.293+0000 7fea37868e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:32.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:52:32.293+0000 7fea37868e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:32.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:52:32.293+0000 7fea37868e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:32.285 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-19T15:52:32.286 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: generated fsid f761dc83-ec01-4a40-8463-2d95add17615 2024-01-19T15:52:32.286 INFO:teuthology.orchestra.run.smithi046.stdout:setting min_mon_release = quincy 2024-01-19T15:52:32.286 INFO:teuthology.orchestra.run.smithi046.stdout:epoch 0 2024-01-19T15:52:32.286 INFO:teuthology.orchestra.run.smithi046.stdout:fsid f761dc83-ec01-4a40-8463-2d95add17615 2024-01-19T15:52:32.286 INFO:teuthology.orchestra.run.smithi046.stdout:last_changed 2024-01-19T15:52:32.294676+0000 2024-01-19T15:52:32.286 INFO:teuthology.orchestra.run.smithi046.stdout:created 2024-01-19T15:52:32.294676+0000 2024-01-19T15:52:32.286 INFO:teuthology.orchestra.run.smithi046.stdout:min_mon_release 17 (quincy) 2024-01-19T15:52:32.286 INFO:teuthology.orchestra.run.smithi046.stdout:election_strategy: 1 2024-01-19T15:52:32.286 INFO:teuthology.orchestra.run.smithi046.stdout:0: [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon.a 2024-01-19T15:52:32.286 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-01-19T15:52:32.287 DEBUG:teuthology.orchestra.run.smithi046:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-19T15:52:32.341 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f761dc83-ec01-4a40-8463-2d95add17615... 2024-01-19T15:52:32.343 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-19T15:52:32.431 INFO:teuthology.orchestra.run.smithi046.stdout:[global] 2024-01-19T15:52:32.431 INFO:teuthology.orchestra.run.smithi046.stdout: chdir = "" 2024-01-19T15:52:32.431 INFO:teuthology.orchestra.run.smithi046.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-19T15:52:32.431 INFO:teuthology.orchestra.run.smithi046.stdout: auth supported = none 2024-01-19T15:52:32.431 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.431 INFO:teuthology.orchestra.run.smithi046.stdout: filestore xattr use omap = true 2024-01-19T15:52:32.431 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.431 INFO:teuthology.orchestra.run.smithi046.stdout: mon clock drift allowed = 1.000 2024-01-19T15:52:32.431 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.431 INFO:teuthology.orchestra.run.smithi046.stdout: osd crush chooseleaf type = 0 2024-01-19T15:52:32.432 INFO:teuthology.orchestra.run.smithi046.stdout: auth debug = true 2024-01-19T15:52:32.432 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.432 INFO:teuthology.orchestra.run.smithi046.stdout: ms die on old message = true 2024-01-19T15:52:32.432 INFO:teuthology.orchestra.run.smithi046.stdout: ms die on bug = true 2024-01-19T15:52:32.432 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.432 INFO:teuthology.orchestra.run.smithi046.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-19T15:52:32.432 INFO:teuthology.orchestra.run.smithi046.stdout: mon pg warn max object skew = 0 2024-01-19T15:52:32.432 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.432 INFO:teuthology.orchestra.run.smithi046.stdout: # disable pg_autoscaler by default for new pools 2024-01-19T15:52:32.432 INFO:teuthology.orchestra.run.smithi046.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-19T15:52:32.432 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.432 INFO:teuthology.orchestra.run.smithi046.stdout: osd pool default size = 2 2024-01-19T15:52:32.432 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.432 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd allow primary affinity = true 2024-01-19T15:52:32.432 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd allow pg remap = true 2024-01-19T15:52:32.433 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on legacy crush tunables = false 2024-01-19T15:52:32.433 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on crush straw calc version zero = false 2024-01-19T15:52:32.433 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on no sortbitwise = false 2024-01-19T15:52:32.433 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on osd down out interval zero = false 2024-01-19T15:52:32.433 INFO:teuthology.orchestra.run.smithi046.stdout: mon warn on too few osds = false 2024-01-19T15:52:32.433 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-19T15:52:32.433 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-19T15:52:32.433 INFO:teuthology.orchestra.run.smithi046.stdout: mon_allow_pool_size_one = true 2024-01-19T15:52:32.433 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.433 INFO:teuthology.orchestra.run.smithi046.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-19T15:52:32.433 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.433 INFO:teuthology.orchestra.run.smithi046.stdout: osd default data pool replay window = 5 2024-01-19T15:52:32.433 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.434 INFO:teuthology.orchestra.run.smithi046.stdout: mon allow pool delete = true 2024-01-19T15:52:32.434 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.434 INFO:teuthology.orchestra.run.smithi046.stdout: mon cluster log file level = debug 2024-01-19T15:52:32.434 INFO:teuthology.orchestra.run.smithi046.stdout: debug asserts on shutdown = true 2024-01-19T15:52:32.434 INFO:teuthology.orchestra.run.smithi046.stdout: mon health detail to clog = false 2024-01-19T15:52:32.434 INFO:teuthology.orchestra.run.smithi046.stdout: mon host = 172.21.15.46 2024-01-19T15:52:32.434 INFO:teuthology.orchestra.run.smithi046.stdout: auth client required = none 2024-01-19T15:52:32.434 INFO:teuthology.orchestra.run.smithi046.stdout: auth cluster required = none 2024-01-19T15:52:32.434 INFO:teuthology.orchestra.run.smithi046.stdout: auth service required = none 2024-01-19T15:52:32.434 INFO:teuthology.orchestra.run.smithi046.stdout: debug asok = 0/0 2024-01-19T15:52:32.434 INFO:teuthology.orchestra.run.smithi046.stdout: debug auth = 0/0 2024-01-19T15:52:32.434 INFO:teuthology.orchestra.run.smithi046.stdout: debug bdev = 0/0 2024-01-19T15:52:32.434 INFO:teuthology.orchestra.run.smithi046.stdout: debug bluefs = 0/0 2024-01-19T15:52:32.434 INFO:teuthology.orchestra.run.smithi046.stdout: debug bluestore = 0/0 2024-01-19T15:52:32.434 INFO:teuthology.orchestra.run.smithi046.stdout: debug buffer = 0/0 2024-01-19T15:52:32.435 INFO:teuthology.orchestra.run.smithi046.stdout: debug civetweb = 0/0 2024-01-19T15:52:32.435 INFO:teuthology.orchestra.run.smithi046.stdout: debug client = 0/0 2024-01-19T15:52:32.435 INFO:teuthology.orchestra.run.smithi046.stdout: debug compressor = 0/0 2024-01-19T15:52:32.435 INFO:teuthology.orchestra.run.smithi046.stdout: debug context = 0/0 2024-01-19T15:52:32.435 INFO:teuthology.orchestra.run.smithi046.stdout: debug crush = 0/0 2024-01-19T15:52:32.435 INFO:teuthology.orchestra.run.smithi046.stdout: debug crypto = 0/0 2024-01-19T15:52:32.435 INFO:teuthology.orchestra.run.smithi046.stdout: debug dpdk = 0/0 2024-01-19T15:52:32.435 INFO:teuthology.orchestra.run.smithi046.stdout: debug eventtrace = 0/0 2024-01-19T15:52:32.435 INFO:teuthology.orchestra.run.smithi046.stdout: debug filer = 0/0 2024-01-19T15:52:32.435 INFO:teuthology.orchestra.run.smithi046.stdout: debug filestore = 0/0 2024-01-19T15:52:32.435 INFO:teuthology.orchestra.run.smithi046.stdout: debug finisher = 0/0 2024-01-19T15:52:32.435 INFO:teuthology.orchestra.run.smithi046.stdout: debug fuse = 0/0 2024-01-19T15:52:32.435 INFO:teuthology.orchestra.run.smithi046.stdout: debug heartbeatmap = 0/0 2024-01-19T15:52:32.435 INFO:teuthology.orchestra.run.smithi046.stdout: debug javaclient = 0/0 2024-01-19T15:52:32.435 INFO:teuthology.orchestra.run.smithi046.stdout: debug journal = 0/0 2024-01-19T15:52:32.436 INFO:teuthology.orchestra.run.smithi046.stdout: debug journaler = 0/0 2024-01-19T15:52:32.436 INFO:teuthology.orchestra.run.smithi046.stdout: debug kstore = 0/0 2024-01-19T15:52:32.436 INFO:teuthology.orchestra.run.smithi046.stdout: debug leveldb = 0/0 2024-01-19T15:52:32.436 INFO:teuthology.orchestra.run.smithi046.stdout: debug lockdep = 0/0 2024-01-19T15:52:32.436 INFO:teuthology.orchestra.run.smithi046.stdout: debug mds = 0/0 2024-01-19T15:52:32.436 INFO:teuthology.orchestra.run.smithi046.stdout: debug mds balancer = 0/0 2024-01-19T15:52:32.436 INFO:teuthology.orchestra.run.smithi046.stdout: debug mds locker = 0/0 2024-01-19T15:52:32.436 INFO:teuthology.orchestra.run.smithi046.stdout: debug mds log = 0/0 2024-01-19T15:52:32.436 INFO:teuthology.orchestra.run.smithi046.stdout: debug mds log expire = 0/0 2024-01-19T15:52:32.436 INFO:teuthology.orchestra.run.smithi046.stdout: debug mds migrator = 0/0 2024-01-19T15:52:32.436 INFO:teuthology.orchestra.run.smithi046.stdout: debug memdb = 0/0 2024-01-19T15:52:32.436 INFO:teuthology.orchestra.run.smithi046.stdout: debug mgr = 0/0 2024-01-19T15:52:32.436 INFO:teuthology.orchestra.run.smithi046.stdout: debug mgrc = 0/0 2024-01-19T15:52:32.436 INFO:teuthology.orchestra.run.smithi046.stdout: debug mon = 0/0 2024-01-19T15:52:32.436 INFO:teuthology.orchestra.run.smithi046.stdout: debug monc = 0/0 2024-01-19T15:52:32.437 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 0/0 2024-01-19T15:52:32.437 INFO:teuthology.orchestra.run.smithi046.stdout: debug objclass = 0/0 2024-01-19T15:52:32.437 INFO:teuthology.orchestra.run.smithi046.stdout: debug objectcacher = 0/0 2024-01-19T15:52:32.437 INFO:teuthology.orchestra.run.smithi046.stdout: debug objecter = 0/0 2024-01-19T15:52:32.437 INFO:teuthology.orchestra.run.smithi046.stdout: debug optracker = 0/0 2024-01-19T15:52:32.437 INFO:teuthology.orchestra.run.smithi046.stdout: debug osd = 0/0 2024-01-19T15:52:32.437 INFO:teuthology.orchestra.run.smithi046.stdout: debug paxos = 0/0 2024-01-19T15:52:32.437 INFO:teuthology.orchestra.run.smithi046.stdout: debug perfcounter = 0/0 2024-01-19T15:52:32.437 INFO:teuthology.orchestra.run.smithi046.stdout: debug rados = 0/0 2024-01-19T15:52:32.437 INFO:teuthology.orchestra.run.smithi046.stdout: debug rbd = 0/0 2024-01-19T15:52:32.437 INFO:teuthology.orchestra.run.smithi046.stdout: debug rbd mirror = 0/0 2024-01-19T15:52:32.437 INFO:teuthology.orchestra.run.smithi046.stdout: debug rbd replay = 0/0 2024-01-19T15:52:32.437 INFO:teuthology.orchestra.run.smithi046.stdout: debug refs = 0/0 2024-01-19T15:52:32.437 INFO:teuthology.orchestra.run.smithi046.stdout: debug rgw = 0/0 2024-01-19T15:52:32.437 INFO:teuthology.orchestra.run.smithi046.stdout: debug rgw sync = 0/0 2024-01-19T15:52:32.438 INFO:teuthology.orchestra.run.smithi046.stdout: debug rocksdb = 0/0 2024-01-19T15:52:32.438 INFO:teuthology.orchestra.run.smithi046.stdout: debug striper = 0/0 2024-01-19T15:52:32.438 INFO:teuthology.orchestra.run.smithi046.stdout: debug throttle = 0/0 2024-01-19T15:52:32.438 INFO:teuthology.orchestra.run.smithi046.stdout: debug timer = 0/0 2024-01-19T15:52:32.438 INFO:teuthology.orchestra.run.smithi046.stdout: debug tp = 0/0 2024-01-19T15:52:32.438 INFO:teuthology.orchestra.run.smithi046.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-01-19T15:52:32.438 INFO:teuthology.orchestra.run.smithi046.stdout: ms client mode = crc 2024-01-19T15:52:32.438 INFO:teuthology.orchestra.run.smithi046.stdout: ms cluster mode = crc 2024-01-19T15:52:32.438 INFO:teuthology.orchestra.run.smithi046.stdout: ms mon client mode = crc 2024-01-19T15:52:32.438 INFO:teuthology.orchestra.run.smithi046.stdout: ms mon cluster mode = crc 2024-01-19T15:52:32.438 INFO:teuthology.orchestra.run.smithi046.stdout: ms mon service mode = crc 2024-01-19T15:52:32.438 INFO:teuthology.orchestra.run.smithi046.stdout: ms service mode = crc 2024-01-19T15:52:32.438 INFO:teuthology.orchestra.run.smithi046.stdout: fsid = f761dc83-ec01-4a40-8463-2d95add17615 2024-01-19T15:52:32.438 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.438 INFO:teuthology.orchestra.run.smithi046.stdout:[osd] 2024-01-19T15:52:32.439 INFO:teuthology.orchestra.run.smithi046.stdout: osd journal size = 100 2024-01-19T15:52:32.439 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.439 INFO:teuthology.orchestra.run.smithi046.stdout: osd scrub load threshold = 5.0 2024-01-19T15:52:32.439 INFO:teuthology.orchestra.run.smithi046.stdout: osd scrub max interval = 600 2024-01-19T15:52:32.439 INFO:teuthology.orchestra.run.smithi046.stdout: osd mclock profile = high_recovery_ops 2024-01-19T15:52:32.439 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.439 INFO:teuthology.orchestra.run.smithi046.stdout: osd recover clone overlap = true 2024-01-19T15:52:32.439 INFO:teuthology.orchestra.run.smithi046.stdout: osd recovery max chunk = 1048576 2024-01-19T15:52:32.439 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.439 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug shutdown = true 2024-01-19T15:52:32.439 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug op order = true 2024-01-19T15:52:32.439 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug verify stray on activate = true 2024-01-19T15:52:32.440 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug trim objects = true 2024-01-19T15:52:32.440 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.440 INFO:teuthology.orchestra.run.smithi046.stdout: osd open classes on start = true 2024-01-19T15:52:32.440 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug pg log writeout = true 2024-01-19T15:52:32.440 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.440 INFO:teuthology.orchestra.run.smithi046.stdout: osd deep scrub update digest min age = 30 2024-01-19T15:52:32.440 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.440 INFO:teuthology.orchestra.run.smithi046.stdout: osd map max advance = 10 2024-01-19T15:52:32.440 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.440 INFO:teuthology.orchestra.run.smithi046.stdout: journal zero on create = true 2024-01-19T15:52:32.440 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.440 INFO:teuthology.orchestra.run.smithi046.stdout: filestore ondisk finisher threads = 3 2024-01-19T15:52:32.440 INFO:teuthology.orchestra.run.smithi046.stdout: filestore apply finisher threads = 3 2024-01-19T15:52:32.440 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.440 INFO:teuthology.orchestra.run.smithi046.stdout: bdev debug aio = true 2024-01-19T15:52:32.441 INFO:teuthology.orchestra.run.smithi046.stdout: osd debug misdirected ops = true 2024-01-19T15:52:32.441 INFO:teuthology.orchestra.run.smithi046.stdout: debug monc = 20 2024-01-19T15:52:32.441 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore block size = 96636764160 2024-01-19T15:52:32.441 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore compression algorithm = snappy 2024-01-19T15:52:32.441 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore compression mode = aggressive 2024-01-19T15:52:32.441 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore fsck on mount = True 2024-01-19T15:52:32.441 INFO:teuthology.orchestra.run.smithi046.stdout: bluestore rocksdb cf = False 2024-01-19T15:52:32.441 INFO:teuthology.orchestra.run.smithi046.stdout: crimson osd obc lru size = 10 2024-01-19T15:52:32.441 INFO:teuthology.orchestra.run.smithi046.stdout: crimson seastar smp = 3 2024-01-19T15:52:32.441 INFO:teuthology.orchestra.run.smithi046.stdout: debug bluefs = 20 2024-01-19T15:52:32.442 INFO:teuthology.orchestra.run.smithi046.stdout: debug bluestore = 20 2024-01-19T15:52:32.442 INFO:teuthology.orchestra.run.smithi046.stdout: debug filestore = 0/0 2024-01-19T15:52:32.442 INFO:teuthology.orchestra.run.smithi046.stdout: debug journal = 0/0 2024-01-19T15:52:32.442 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 0/0 2024-01-19T15:52:32.442 INFO:teuthology.orchestra.run.smithi046.stdout: debug osd = 0/0 2024-01-19T15:52:32.442 INFO:teuthology.orchestra.run.smithi046.stdout: debug rocksdb = 10 2024-01-19T15:52:32.442 INFO:teuthology.orchestra.run.smithi046.stdout: err to stderr = True 2024-01-19T15:52:32.442 INFO:teuthology.orchestra.run.smithi046.stdout: log flush on exit = True 2024-01-19T15:52:32.442 INFO:teuthology.orchestra.run.smithi046.stdout: log to file = False 2024-01-19T15:52:32.442 INFO:teuthology.orchestra.run.smithi046.stdout: log to stderr = True 2024-01-19T15:52:32.442 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd backfillfull_ratio = 0.85 2024-01-19T15:52:32.442 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd full ratio = 0.9 2024-01-19T15:52:32.442 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd nearfull ratio = 0.8 2024-01-19T15:52:32.443 INFO:teuthology.orchestra.run.smithi046.stdout: osd failsafe full ratio = 0.95 2024-01-19T15:52:32.443 INFO:teuthology.orchestra.run.smithi046.stdout: osd objectstore = bluestore 2024-01-19T15:52:32.443 INFO:teuthology.orchestra.run.smithi046.stdout: osd shutdown pgref assert = True 2024-01-19T15:52:32.443 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.443 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr] 2024-01-19T15:52:32.443 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 1 2024-01-19T15:52:32.443 INFO:teuthology.orchestra.run.smithi046.stdout: debug mgr = 20 2024-01-19T15:52:32.443 INFO:teuthology.orchestra.run.smithi046.stdout: debug mon = 20 2024-01-19T15:52:32.443 INFO:teuthology.orchestra.run.smithi046.stdout: debug auth = 20 2024-01-19T15:52:32.443 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min pgs per osd = 4 2024-01-19T15:52:32.443 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min bytes per osd = 10 2024-01-19T15:52:32.443 INFO:teuthology.orchestra.run.smithi046.stdout: mgr/telemetry/nag = false 2024-01-19T15:52:32.443 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.443 INFO:teuthology.orchestra.run.smithi046.stdout:[mon] 2024-01-19T15:52:32.443 INFO:teuthology.orchestra.run.smithi046.stdout: debug ms = 0/0 2024-01-19T15:52:32.443 INFO:teuthology.orchestra.run.smithi046.stdout: debug mon = 0/0 2024-01-19T15:52:32.444 INFO:teuthology.orchestra.run.smithi046.stdout: debug paxos = 0/0 2024-01-19T15:52:32.444 INFO:teuthology.orchestra.run.smithi046.stdout: debug auth = 20 2024-01-19T15:52:32.444 INFO:teuthology.orchestra.run.smithi046.stdout: mon data avail warn = 5 2024-01-19T15:52:32.444 INFO:teuthology.orchestra.run.smithi046.stdout: mon mgr mkfs grace = 240 2024-01-19T15:52:32.444 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min pgs per osd = 4 2024-01-19T15:52:32.444 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd reporter subtree level = osd 2024-01-19T15:52:32.444 INFO:teuthology.orchestra.run.smithi046.stdout: mon osd prime pg temp = true 2024-01-19T15:52:32.444 INFO:teuthology.orchestra.run.smithi046.stdout: mon reweight min bytes per osd = 10 2024-01-19T15:52:32.444 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.444 INFO:teuthology.orchestra.run.smithi046.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-19T15:52:32.444 INFO:teuthology.orchestra.run.smithi046.stdout: auth mon ticket ttl = 660 # 11m 2024-01-19T15:52:32.444 INFO:teuthology.orchestra.run.smithi046.stdout: auth service ticket ttl = 240 # 4m 2024-01-19T15:52:32.445 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.445 INFO:teuthology.orchestra.run.smithi046.stdout: # don't complain about insecure global_id in the test suite 2024-01-19T15:52:32.445 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-19T15:52:32.445 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-19T15:52:32.445 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.445 INFO:teuthology.orchestra.run.smithi046.stdout: # 1m isn't quite enough 2024-01-19T15:52:32.445 INFO:teuthology.orchestra.run.smithi046.stdout: mon_down_mkfs_grace = 2m 2024-01-19T15:52:32.445 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.445 INFO:teuthology.orchestra.run.smithi046.stdout: mon_warn_on_filestore_osds = false 2024-01-19T15:52:32.445 INFO:teuthology.orchestra.run.smithi046.stdout: osd pool default crimson = True 2024-01-19T15:52:32.445 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:32.445 INFO:teuthology.orchestra.run.smithi046.stdout:[client] 2024-01-19T15:52:32.445 INFO:teuthology.orchestra.run.smithi046.stdout: rgw cache enabled = true 2024-01-19T15:52:32.445 INFO:teuthology.orchestra.run.smithi046.stdout: rgw enable ops log = true 2024-01-19T15:52:32.446 INFO:teuthology.orchestra.run.smithi046.stdout: rgw enable usage log = true 2024-01-19T15:52:32.446 INFO:teuthology.orchestra.run.smithi046.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-19T15:52:32.446 INFO:teuthology.orchestra.run.smithi046.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-19T15:52:32.446 INFO:teuthology.orchestra.run.smithi046.stdout:[mon.a] 2024-01-19T15:52:32.447 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-19T15:52:32.447 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-19T15:52:32.507 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-19T15:52:32.508 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-19T15:52:32.508 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-19T15:52:32.525 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-19T15:52:32.526 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-19T15:52:32.579 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:52:32.580 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-19T15:52:32.580 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-19T15:52:32.580 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-19T15:52:32.658 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-19T15:52:32.658 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-19T15:52:32.713 INFO:tasks.ceph:Setting up mon nodes... 2024-01-19T15:52:32.713 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-19T15:52:32.713 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-19T15:52:32.814 INFO:teuthology.orchestra.run.smithi046.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-19T15:52:32.816 INFO:tasks.ceph:Setting up mds nodes... 2024-01-19T15:52:32.816 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-19T15:52:32.817 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-19T15:52:32.864 INFO:teuthology.orchestra.run.smithi046.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-19T15:52:32.879 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-19T15:52:32.879 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi046.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-01-19T15:52:32.879 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-19T15:52:32.906 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-19T15:52:32.906 INFO:tasks.ceph:role: osd.0 2024-01-19T15:52:32.906 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:52:32.906 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-19T15:52:32.972 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-19T15:52:32.972 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-19T15:52:32.972 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-19T15:52:32.972 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-19T15:52:32.972 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-19T15:52:32.972 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-01-19T15:52:32.972 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-19T15:52:32.973 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-19T15:52:32.973 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-19T15:52:32.973 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-19T15:52:33.005 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-01-19T15:52:33.007 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-01-19T15:52:33.007 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-19T15:52:33.078 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-19T15:52:33.149 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-19T15:52:33.216 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-19T15:52:33.216 INFO:tasks.ceph:role: osd.1 2024-01-19T15:52:33.216 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:52:33.216 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-19T15:52:33.282 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-19T15:52:33.283 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-19T15:52:33.283 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-19T15:52:33.283 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-19T15:52:33.283 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-19T15:52:33.283 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-01-19T15:52:33.283 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-19T15:52:33.283 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-19T15:52:33.283 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-19T15:52:33.283 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-19T15:52:33.315 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-01-19T15:52:33.317 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-01-19T15:52:33.317 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-19T15:52:33.387 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-19T15:52:33.457 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-19T15:52:33.524 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-19T15:52:33.524 INFO:tasks.ceph:role: osd.2 2024-01-19T15:52:33.524 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:52:33.524 DEBUG:teuthology.orchestra.run.smithi046:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-19T15:52:33.590 INFO:teuthology.orchestra.run.smithi046.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-19T15:52:33.590 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-19T15:52:33.590 INFO:teuthology.orchestra.run.smithi046.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-19T15:52:33.590 INFO:teuthology.orchestra.run.smithi046.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-19T15:52:33.590 INFO:teuthology.orchestra.run.smithi046.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-19T15:52:33.590 INFO:teuthology.orchestra.run.smithi046.stdout: = sunit=0 swidth=0 blks 2024-01-19T15:52:33.590 INFO:teuthology.orchestra.run.smithi046.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-19T15:52:33.590 INFO:teuthology.orchestra.run.smithi046.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-19T15:52:33.590 INFO:teuthology.orchestra.run.smithi046.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-19T15:52:33.590 INFO:teuthology.orchestra.run.smithi046.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-19T15:52:33.622 INFO:teuthology.orchestra.run.smithi046.stdout:Discarding blocks...Done. 2024-01-19T15:52:33.623 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi046.front.sepia.ceph.com -o noatime 2024-01-19T15:52:33.624 DEBUG:teuthology.orchestra.run.smithi046:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-19T15:52:33.693 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-19T15:52:33.769 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-19T15:52:34.302 INFO:teuthology.orchestra.run.smithi046.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-19T15:52:34.303 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:34,313 seastar - Reactor backend: linux-aio 2024-01-19T15:52:34.319 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:34,328 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-19T15:52:34.319 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:34,328 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-19T15:52:34.319 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:34,328 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-19T15:52:34.319 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:34,328 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-19T15:52:34.321 INFO:teuthology.orchestra.run.smithi046.stderr:==28371==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-19T15:52:34.327 INFO:teuthology.orchestra.run.smithi046.stderr:==28371==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc56d6e000; bottom 0x7f0a84e55000; size: 0x00f1d1f19000 (1038609387520) 2024-01-19T15:52:34.327 INFO:teuthology.orchestra.run.smithi046.stderr:False positive error reports may follow 2024-01-19T15:52:34.327 INFO:teuthology.orchestra.run.smithi046.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-19T15:52:34.886 INFO:teuthology.orchestra.run.smithi046.stderr:----------------------------------------------------- 2024-01-19T15:52:34.886 INFO:teuthology.orchestra.run.smithi046.stderr:Suppressions used: 2024-01-19T15:52:34.886 INFO:teuthology.orchestra.run.smithi046.stderr: count bytes template 2024-01-19T15:52:34.886 INFO:teuthology.orchestra.run.smithi046.stderr: 1 8 MallocExtension::Register 2024-01-19T15:52:34.886 INFO:teuthology.orchestra.run.smithi046.stderr:----------------------------------------------------- 2024-01-19T15:52:34.886 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-01-19T15:52:34.950 INFO:teuthology.orchestra.run.smithi046.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-19T15:52:34.950 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:34,960 seastar - Reactor backend: linux-aio 2024-01-19T15:52:34.968 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:34,977 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-19T15:52:34.968 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:34,978 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-19T15:52:34.968 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:34,978 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-19T15:52:34.976 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:34,985 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-19T15:52:34.978 INFO:teuthology.orchestra.run.smithi046.stderr:==28369==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-19T15:52:34.984 INFO:teuthology.orchestra.run.smithi046.stderr:==28369==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffc56d6e000; bottom 0x7f0a82c3e000; size: 0x00f1d4130000 (1038645133312) 2024-01-19T15:52:34.984 INFO:teuthology.orchestra.run.smithi046.stderr:False positive error reports may follow 2024-01-19T15:52:34.984 INFO:teuthology.orchestra.run.smithi046.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-19T15:52:39.007 INFO:teuthology.orchestra.run.smithi046.stderr:----------------------------------------------------- 2024-01-19T15:52:39.007 INFO:teuthology.orchestra.run.smithi046.stderr:Suppressions used: 2024-01-19T15:52:39.007 INFO:teuthology.orchestra.run.smithi046.stderr: count bytes template 2024-01-19T15:52:39.007 INFO:teuthology.orchestra.run.smithi046.stderr: 1 8 MallocExtension::Register 2024-01-19T15:52:39.007 INFO:teuthology.orchestra.run.smithi046.stderr:----------------------------------------------------- 2024-01-19T15:52:39.008 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-01-19T15:52:39.008 INFO:teuthology.orchestra.run.smithi046.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid f761dc83-ec01-4a40-8463-2d95add17615 2024-01-19T15:52:39.047 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-19T15:52:39.073 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-19T15:52:39.599 INFO:teuthology.orchestra.run.smithi046.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-19T15:52:39.600 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:39,609 seastar - Reactor backend: linux-aio 2024-01-19T15:52:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:39,625 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-19T15:52:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:39,625 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-19T15:52:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:39,625 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-19T15:52:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:39,625 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-19T15:52:39.618 INFO:teuthology.orchestra.run.smithi046.stderr:==28553==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-19T15:52:39.624 INFO:teuthology.orchestra.run.smithi046.stderr:==28553==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd50597000; bottom 0x7fca66489000; size: 0x0032ea10e000 (218675339264) 2024-01-19T15:52:39.624 INFO:teuthology.orchestra.run.smithi046.stderr:False positive error reports may follow 2024-01-19T15:52:39.624 INFO:teuthology.orchestra.run.smithi046.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-19T15:52:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:----------------------------------------------------- 2024-01-19T15:52:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:Suppressions used: 2024-01-19T15:52:40.152 INFO:teuthology.orchestra.run.smithi046.stderr: count bytes template 2024-01-19T15:52:40.152 INFO:teuthology.orchestra.run.smithi046.stderr: 1 8 MallocExtension::Register 2024-01-19T15:52:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:----------------------------------------------------- 2024-01-19T15:52:40.152 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-01-19T15:52:40.206 INFO:teuthology.orchestra.run.smithi046.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-19T15:52:40.206 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:40,216 seastar - Reactor backend: linux-aio 2024-01-19T15:52:40.234 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:40,243 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-19T15:52:40.234 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:40,243 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-19T15:52:40.234 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:40,243 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-19T15:52:40.241 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:40,251 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-19T15:52:40.244 INFO:teuthology.orchestra.run.smithi046.stderr:==28551==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-19T15:52:40.250 INFO:teuthology.orchestra.run.smithi046.stderr:==28551==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffd50597000; bottom 0x7fca64272000; size: 0x0032ec325000 (218711085056) 2024-01-19T15:52:40.250 INFO:teuthology.orchestra.run.smithi046.stderr:False positive error reports may follow 2024-01-19T15:52:40.251 INFO:teuthology.orchestra.run.smithi046.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-19T15:52:40.276 INFO:teuthology.orchestra.run.smithi046.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2e81660e 0x2e8179bc 0x2e818eee 0x2e819272 0x2e8193bc 0x2e819812 0x54daf 0x23f791b1 0x23f80d29 0x23f85e4b 0x23f8b4bc 0x25eb8564 0x25ebf5e2 0x25ec1f2f 0x25ec30f4 0x23ace913 0x23aeb87a 0x23b60925 0x2e788fc7 0x2e7ec1f8 0x2ea921fd 0x2ea94047 0x2e4454a4 0x2e4473ad 0x1f716a88 0x3feaf 0x3ff5f 0x1f2d5b64 2024-01-19T15:52:40.276 INFO:teuthology.orchestra.run.smithi046.stderr:kernel callstack: 2024-01-19T15:52:44.281 INFO:teuthology.orchestra.run.smithi046.stderr:----------------------------------------------------- 2024-01-19T15:52:44.281 INFO:teuthology.orchestra.run.smithi046.stderr:Suppressions used: 2024-01-19T15:52:44.281 INFO:teuthology.orchestra.run.smithi046.stderr: count bytes template 2024-01-19T15:52:44.281 INFO:teuthology.orchestra.run.smithi046.stderr: 1 8 MallocExtension::Register 2024-01-19T15:52:44.281 INFO:teuthology.orchestra.run.smithi046.stderr:----------------------------------------------------- 2024-01-19T15:52:44.282 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-01-19T15:52:44.282 INFO:teuthology.orchestra.run.smithi046.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid f761dc83-ec01-4a40-8463-2d95add17615 2024-01-19T15:52:44.320 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-19T15:52:44.347 DEBUG:teuthology.orchestra.run.smithi046:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-19T15:52:44.872 INFO:teuthology.orchestra.run.smithi046.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-19T15:52:44.873 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:44,882 seastar - Reactor backend: linux-aio 2024-01-19T15:52:44.889 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:44,898 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-19T15:52:44.890 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:44,898 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-19T15:52:44.890 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:44,898 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-19T15:52:44.890 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:44,898 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-19T15:52:44.891 INFO:teuthology.orchestra.run.smithi046.stderr:==28735==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-19T15:52:44.897 INFO:teuthology.orchestra.run.smithi046.stderr:==28735==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe7a6f0000; bottom 0x7fc7af2fa000; size: 0x0036cb3f6000 (235338162176) 2024-01-19T15:52:44.897 INFO:teuthology.orchestra.run.smithi046.stderr:False positive error reports may follow 2024-01-19T15:52:44.897 INFO:teuthology.orchestra.run.smithi046.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-19T15:52:44.923 INFO:teuthology.orchestra.run.smithi046.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2e81660e 0x2e8179bc 0x2e818eee 0x2e819272 0x2e8193bc 0x2e819812 0x54daf 0x24156f83 0x24159a31 0x2414e94d 0x241507d7 0x2416a8be 0x2416d7f9 0x2417ff20 0x241abc37 0x241ac4a3 0x2415fd97 0x2416f1e4 0x241708c3 0x24171298 0x2417235e 0x24187194 0x24187925 0x24189854 0x2418b258 0x2418bfb6 0x2414c425 0x2414ba94 0x23ff022b 0x25eb7aae 0x25ebf5e2 0x25ec1f2f 0x25ec30f4 0x23ace913 0x23aeb87a 0x23b60925 0x2e788fc7 0x2e7ec1f8 0x2ea921fd 0x2ea94047 0x2e4454a4 0x2e4473ad 0x1f937895 0x1f93cf5b 0x1f7151d2 0x3feaf 0x3ff5f 0x1f2d5b64 2024-01-19T15:52:44.923 INFO:teuthology.orchestra.run.smithi046.stderr:kernel callstack: 2024-01-19T15:52:45.427 INFO:teuthology.orchestra.run.smithi046.stderr:----------------------------------------------------- 2024-01-19T15:52:45.427 INFO:teuthology.orchestra.run.smithi046.stderr:Suppressions used: 2024-01-19T15:52:45.427 INFO:teuthology.orchestra.run.smithi046.stderr: count bytes template 2024-01-19T15:52:45.427 INFO:teuthology.orchestra.run.smithi046.stderr: 1 8 MallocExtension::Register 2024-01-19T15:52:45.428 INFO:teuthology.orchestra.run.smithi046.stderr:----------------------------------------------------- 2024-01-19T15:52:45.428 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-01-19T15:52:45.486 INFO:teuthology.orchestra.run.smithi046.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-19T15:52:45.486 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:45,496 seastar - Reactor backend: linux-aio 2024-01-19T15:52:45.515 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:45,523 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-19T15:52:45.515 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:45,523 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-19T15:52:45.515 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:45,523 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-19T15:52:45.521 INFO:teuthology.orchestra.run.smithi046.stderr:INFO 2024-01-19 15:52:45,531 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-19T15:52:45.524 INFO:teuthology.orchestra.run.smithi046.stderr:==28733==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-19T15:52:45.530 INFO:teuthology.orchestra.run.smithi046.stderr:==28733==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffe7a6f0000; bottom 0x7fc7ad0e3000; size: 0x0036cd60d000 (235373907968) 2024-01-19T15:52:45.530 INFO:teuthology.orchestra.run.smithi046.stderr:False positive error reports may follow 2024-01-19T15:52:45.530 INFO:teuthology.orchestra.run.smithi046.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-19T15:52:45.556 INFO:teuthology.orchestra.run.smithi046.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2e81660e 0x2e8179bc 0x2e818eee 0x2e819272 0x2e8193bc 0x2e819812 0x54daf 0x23eeb588 0x23f80c59 0x23f85e4b 0x23f8b4bc 0x25eb8564 0x25ebf5e2 0x25ec1f2f 0x25ec30f4 0x23ace913 0x23aeb87a 0x23b60925 0x2e788fc7 0x2e7ec1f8 0x2ea921fd 0x2ea94047 0x2e4454a4 0x2e4473ad 0x1f716a88 0x3feaf 0x3ff5f 0x1f2d5b64 2024-01-19T15:52:45.556 INFO:teuthology.orchestra.run.smithi046.stderr:kernel callstack: 2024-01-19T15:52:49.561 INFO:teuthology.orchestra.run.smithi046.stderr:----------------------------------------------------- 2024-01-19T15:52:49.561 INFO:teuthology.orchestra.run.smithi046.stderr:Suppressions used: 2024-01-19T15:52:49.561 INFO:teuthology.orchestra.run.smithi046.stderr: count bytes template 2024-01-19T15:52:49.561 INFO:teuthology.orchestra.run.smithi046.stderr: 1 8 MallocExtension::Register 2024-01-19T15:52:49.561 INFO:teuthology.orchestra.run.smithi046.stderr:----------------------------------------------------- 2024-01-19T15:52:49.562 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-01-19T15:52:49.562 INFO:teuthology.orchestra.run.smithi046.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid f761dc83-ec01-4a40-8463-2d95add17615 2024-01-19T15:52:49.602 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-19T15:52:49.628 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-19T15:52:49.629 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-19T15:52:49.629 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-19T15:52:49.696 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-19T15:52:49.696 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-19T15:52:49.763 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-19T15:52:49.763 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-19T15:52:49.830 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-19T15:52:49.830 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-19T15:52:49.897 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-19T15:52:49.897 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-19T15:52:49.952 INFO:tasks.ceph:Adding keys to all mons... 2024-01-19T15:52:49.952 DEBUG:teuthology.orchestra.run.smithi046:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-19T15:52:50.017 INFO:teuthology.orchestra.run.smithi046.stdout:[mgr.x] 2024-01-19T15:52:50.017 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQDAmqplPWYTMRAAhlb0syVi3xksanIWVxD07Q== 2024-01-19T15:52:50.017 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.0] 2024-01-19T15:52:50.017 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQDDmqpl+EYaAxAAnSKbFKCwjs8WbS9aTTeV3w== 2024-01-19T15:52:50.017 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.1] 2024-01-19T15:52:50.017 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQDImqpl1+djEhAATFFCnKrFf42bjNGjRvWEzA== 2024-01-19T15:52:50.017 INFO:teuthology.orchestra.run.smithi046.stdout:[osd.2] 2024-01-19T15:52:50.017 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQDNmqplRlUQIxAATKXUD4UqfY1mjF79WxFPBQ== 2024-01-19T15:52:50.017 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2024-01-19T15:52:50.018 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQDAmqpl9JIONBAATFJvkZKVomEW9RL2KfbiIw== 2024-01-19T15:52:50.018 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-19T15:52:50.115 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-19T15:52:50.172 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-19T15:52:50.229 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-19T15:52:50.286 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-19T15:52:50.346 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-19T15:52:50.346 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-19T15:52:50.372 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-19T15:52:50.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:52:50.482+0000 7f6a60c829c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:50.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:52:50.482+0000 7f6a60c829c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:50.485 DEBUG:teuthology.orchestra.run.smithi046:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-19T15:52:50.512 DEBUG:teuthology.orchestra.run.smithi046:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-19T15:52:50.567 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-19T15:52:50.567 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-19T15:52:50.567 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-01-19T15:52:50.610 INFO:tasks.ceph.mon.a:Started 2024-01-19T15:52:50.610 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-19T15:52:50.610 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-19T15:52:50.610 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-01-19T15:52:50.612 INFO:tasks.ceph.mgr.x:Started 2024-01-19T15:52:50.612 INFO:tasks.ceph:Setting crush tunables to default 2024-01-19T15:52:50.612 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd crush tunables default 2024-01-19T15:52:50.690 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-01-19T15:52:50.698+0000 7f79e785e9c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:50.690 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-01-19T15:52:50.698+0000 7f79e785e9c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:50.690 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-01-19T15:52:50.698+0000 7f79e785e9c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:50.692 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:50.701+0000 7ff36876bf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:50.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:52:50.724+0000 7f1771f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:50.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:52:50.741+0000 7f1771f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:50.804 INFO:teuthology.orchestra.run.smithi046.stderr:adjusted tunables profile to default 2024-01-19T15:52:50.814 INFO:tasks.ceph:check_enable_crimson: True 2024-01-19T15:52:50.815 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-01-19T15:52:50.815 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-01-19T15:52:50.895 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:50.903+0000 7ff36876bf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:50.895 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:50.903+0000 7ff36876bf00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:50.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:52:50.930+0000 7f158c19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:50.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:52:50.936+0000 7f158c19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:51.008 INFO:tasks.ceph.mon.a.smithi046.stderr:2024-01-19T15:52:51.016+0000 7f79df7f1640 -1 WARNING: experimental feature 'crimson' is enabled 2024-01-19T15:52:51.008 INFO:tasks.ceph.mon.a.smithi046.stderr:Please be aware that this feature is experimental, untested, 2024-01-19T15:52:51.008 INFO:tasks.ceph.mon.a.smithi046.stderr:unsupported, and may result in data corruption, data loss, 2024-01-19T15:52:51.008 INFO:tasks.ceph.mon.a.smithi046.stderr:and/or irreparable damage to your cluster. Do not use 2024-01-19T15:52:51.008 INFO:tasks.ceph.mon.a.smithi046.stderr:feature with important data. 2024-01-19T15:52:51.008 INFO:tasks.ceph.mon.a.smithi046.stderr: 2024-01-19T15:52:51.369 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:51.378+0000 7ff36876bf00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-19T15:52:51.424 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:51.433+0000 7ff36876bf00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-19T15:52:51.819 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-19T15:52:51.819 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-19T15:52:51.820 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-19T15:52:51.846 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-19T15:52:51.846 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-19T15:52:51.912 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-19T15:52:51.913 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-19T15:52:51.972 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:51.980+0000 7ff36876bf00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-19T15:52:51.980 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd new 7edd3b65-af23-4405-8062-92a7a0ba507a 0 2024-01-19T15:52:52.027 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:52.035+0000 7ff36876bf00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-19T15:52:52.074 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:52.083+0000 7ff36876bf00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-19T15:52:52.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:52:52.130+0000 7f0a1a6de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:52.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:52:52.137+0000 7f0a1a6de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:52.132 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:52.141+0000 7ff36876bf00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-19T15:52:52.188 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:52.196+0000 7ff36876bf00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-19T15:52:52.222 INFO:teuthology.orchestra.run.smithi046.stdout:0 2024-01-19T15:52:52.234 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd new 56141df5-c934-4679-9d66-cc6f083470eb 1 2024-01-19T15:52:52.296 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:52.305+0000 7ff36876bf00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-19T15:52:52.371 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:52.380+0000 7ff36876bf00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-19T15:52:52.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:52:52.384+0000 7fcc20e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:52.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:52:52.390+0000 7fcc20e6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:52.419 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:52.428+0000 7ff36876bf00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-19T15:52:52.459 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-01-19T15:52:52.470 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd new 43f23619-ba49-4dda-b0a1-2a6578732b13 2 2024-01-19T15:52:52.474 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:52.483+0000 7ff36876bf00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-19T15:52:52.524 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:52.533+0000 7ff36876bf00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-19T15:52:52.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:52:52.583+0000 7f76c1d04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:52.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:52:52.589+0000 7f76c1d04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:52.659 INFO:teuthology.orchestra.run.smithi046.stdout:2 2024-01-19T15:52:52.669 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-01-19T15:52:52.700 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-19T15:52:52.701 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-01-19T15:52:52.742 INFO:tasks.ceph.osd.0:Started 2024-01-19T15:52:52.743 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-01-19T15:52:52.773 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-19T15:52:52.773 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-01-19T15:52:52.816 INFO:tasks.ceph.osd.1:Started 2024-01-19T15:52:52.816 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-01-19T15:52:52.849 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-19T15:52:52.849 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-01-19T15:52:52.893 INFO:tasks.ceph.osd.2:Started 2024-01-19T15:52:52.893 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-19T15:52:52.959 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:52.967+0000 7ff36876bf00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-19T15:52:53.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:52:53.095+0000 7f20bec64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:53.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:52:53.106+0000 7f20bec64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:53.176 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:53.184+0000 7ff36876bf00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-19T15:52:53.230 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:52:53.230 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":6,"fsid":"f761dc83-ec01-4a40-8463-2d95add17615","created":"2024-01-19T15:52:50.735967+0000","modified":"2024-01-19T15:52:52.667772+0000","last_up_change":"0.000000","last_in_change":"2024-01-19T15:52:52.667772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"7edd3b65-af23-4405-8062-92a7a0ba507a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"56141df5-c934-4679-9d66-cc6f083470eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"43f23619-ba49-4dda-b0a1-2a6578732b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-19T15:52:53.252 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-19T15:52:53.253 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-19T15:52:53.320 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:53.329+0000 7ff36876bf00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-19T15:52:53.371 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:53.379+0000 7ff36876bf00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-19T15:52:53.494 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:53.502+0000 7ff36876bf00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-19T15:52:53.555 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-19T15:52:53.596 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:53.605+0000 7ff36876bf00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-19T15:52:53.654 INFO:teuthology.misc.health.smithi046.stderr:2024-01-19T15:52:53.659+0000 7f37bf26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:53.656 INFO:teuthology.misc.health.smithi046.stderr:2024-01-19T15:52:53.665+0000 7f37bf26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:52:53.723 INFO:teuthology.misc.health.smithi046.stdout: 2024-01-19T15:52:53.724 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":6,"fsid":"f761dc83-ec01-4a40-8463-2d95add17615","created":"2024-01-19T15:52:50.735967+0000","modified":"2024-01-19T15:52:52.667772+0000","last_up_change":"0.000000","last_in_change":"2024-01-19T15:52:52.667772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"7edd3b65-af23-4405-8062-92a7a0ba507a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"56141df5-c934-4679-9d66-cc6f083470eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"43f23619-ba49-4dda-b0a1-2a6578732b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-19T15:52:53.734 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-01-19T15:52:53.752 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:53.761+0000 7ff36876bf00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-19T15:52:53.800 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:53.809+0000 7ff36876bf00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-19T15:52:53.954 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:53.962+0000 7ff36876bf00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-19T15:52:54.817 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:54.825+0000 7ff36876bf00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-19T15:52:55.115 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:55.123+0000 7ff36876bf00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-19T15:52:55.170 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:55.178+0000 7ff36876bf00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-19T15:52:55.279 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:55.287+0000 7ff36876bf00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-19T15:52:56.299 INFO:tasks.ceph.mgr.x.smithi046.stderr:2024-01-19T15:52:56.307+0000 7ff30853a640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-01-19T15:53:00.037 DEBUG:teuthology.orchestra.run.smithi046:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-19T15:53:00.147 INFO:teuthology.misc.health.smithi046.stderr:2024-01-19T15:53:00.152+0000 7faf18d1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:00.148 INFO:teuthology.misc.health.smithi046.stderr:2024-01-19T15:53:00.157+0000 7faf18d1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:00.335 INFO:teuthology.misc.health.smithi046.stdout: 2024-01-19T15:53:00.335 INFO:teuthology.misc.health.smithi046.stdout:{"epoch":9,"fsid":"f761dc83-ec01-4a40-8463-2d95add17615","created":"2024-01-19T15:52:50.735967+0000","modified":"2024-01-19T15:53:00.311463+0000","last_up_change":"2024-01-19T15:52:58.307526+0000","last_in_change":"2024-01-19T15:52:52.667772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-19T15:52:59.346560+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7edd3b65-af23-4405-8062-92a7a0ba507a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6807","nonce":420701529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":420701529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":420701529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6813","nonce":420701529}]},"public_addr":"172.21.15.46:6807/420701529","cluster_addr":"172.21.15.46:6806/420701529","heartbeat_back_addr":"172.21.15.46:6812/420701529","heartbeat_front_addr":"172.21.15.46:6813/420701529","state":["exists","up"]},{"osd":1,"uuid":"56141df5-c934-4679-9d66-cc6f083470eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2306118426}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6805","nonce":2306118426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6809","nonce":2306118426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2306118426}]},"public_addr":"172.21.15.46:6804/2306118426","cluster_addr":"172.21.15.46:6805/2306118426","heartbeat_back_addr":"172.21.15.46:6809/2306118426","heartbeat_front_addr":"172.21.15.46:6808/2306118426","state":["exists","up"]},{"osd":2,"uuid":"43f23619-ba49-4dda-b0a1-2a6578732b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6803","nonce":3493997451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3493997451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3493997451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6811","nonce":3493997451}]},"public_addr":"172.21.15.46:6803/3493997451","cluster_addr":"172.21.15.46:6802/3493997451","heartbeat_back_addr":"172.21.15.46:6810/3493997451","heartbeat_front_addr":"172.21.15.46:6811/3493997451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-19T15:53:00.345 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-01-19T15:53:00.345 INFO:tasks.ceph:Creating RBD pool 2024-01-19T15:53:00.346 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd pool create rbd 8 2024-01-19T15:53:00.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:00.454+0000 7fd45106d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:00.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:00.461+0000 7fd45106d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:01.305 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'rbd' created 2024-01-19T15:53:01.322 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-01-19T15:53:01.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:01.448+0000 7fdb0faad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:01.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:01.459+0000 7fdb0faad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:02.308 INFO:teuthology.orchestra.run.smithi046.stderr:enabled application 'rbd' on pool 'rbd' 2024-01-19T15:53:02.322 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-19T15:53:02.323 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-19T15:53:02.323 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-19T15:53:02.323 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-19T15:53:02.323 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-19T15:53:02.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:02.453+0000 7f4239f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:02.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:02.459+0000 7f4239f5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:02.684 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:53:02.694 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":4,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":3157541146},{"type":"v1","addr":"172.21.15.46:6801","nonce":3157541146}]},"active_addr":"172.21.15.46:6801/3157541146","active_change":"2024-01-19T15:52:55.301848+0000","active_mgr_features":4540701272860131327,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":1313928148}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":245298612}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":3594434886}]}]} 2024-01-19T15:53:02.698 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-19T15:53:02.698 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-19T15:53:02.698 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-19T15:53:02.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:02.810+0000 7f833ee6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:02.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:02.815+0000 7f833ee6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:02.992 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:53:02.992 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":11,"fsid":"f761dc83-ec01-4a40-8463-2d95add17615","created":"2024-01-19T15:52:50.735967+0000","modified":"2024-01-19T15:53:02.317368+0000","last_up_change":"2024-01-19T15:52:58.307526+0000","last_in_change":"2024-01-19T15:52:52.667772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-19T15:52:59.346560+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-19T15:53:00.664324+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7edd3b65-af23-4405-8062-92a7a0ba507a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6807","nonce":420701529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":420701529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":420701529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6813","nonce":420701529}]},"public_addr":"172.21.15.46:6807/420701529","cluster_addr":"172.21.15.46:6806/420701529","heartbeat_back_addr":"172.21.15.46:6812/420701529","heartbeat_front_addr":"172.21.15.46:6813/420701529","state":["exists","up"]},{"osd":1,"uuid":"56141df5-c934-4679-9d66-cc6f083470eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2306118426}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6805","nonce":2306118426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6809","nonce":2306118426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2306118426}]},"public_addr":"172.21.15.46:6804/2306118426","cluster_addr":"172.21.15.46:6805/2306118426","heartbeat_back_addr":"172.21.15.46:6809/2306118426","heartbeat_front_addr":"172.21.15.46:6808/2306118426","state":["exists","up"]},{"osd":2,"uuid":"43f23619-ba49-4dda-b0a1-2a6578732b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6803","nonce":3493997451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3493997451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3493997451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6811","nonce":3493997451}]},"public_addr":"172.21.15.46:6803/3493997451","cluster_addr":"172.21.15.46:6802/3493997451","heartbeat_back_addr":"172.21.15.46:6810/3493997451","heartbeat_front_addr":"172.21.15.46:6811/3493997451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-19T15:53:03.004 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-19T15:53:03.004 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-19T15:53:03.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:03.116+0000 7f362b22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:03.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:03.121+0000 7f362b22d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:03.301 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:53:03.301 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":11,"fsid":"f761dc83-ec01-4a40-8463-2d95add17615","created":"2024-01-19T15:52:50.735967+0000","modified":"2024-01-19T15:53:02.317368+0000","last_up_change":"2024-01-19T15:52:58.307526+0000","last_in_change":"2024-01-19T15:52:52.667772+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-19T15:52:59.346560+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-19T15:53:00.664324+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7edd3b65-af23-4405-8062-92a7a0ba507a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6807","nonce":420701529}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":420701529}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":420701529}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6813","nonce":420701529}]},"public_addr":"172.21.15.46:6807/420701529","cluster_addr":"172.21.15.46:6806/420701529","heartbeat_back_addr":"172.21.15.46:6812/420701529","heartbeat_front_addr":"172.21.15.46:6813/420701529","state":["exists","up"]},{"osd":1,"uuid":"56141df5-c934-4679-9d66-cc6f083470eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":2306118426}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6805","nonce":2306118426}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6809","nonce":2306118426}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":2306118426}]},"public_addr":"172.21.15.46:6804/2306118426","cluster_addr":"172.21.15.46:6805/2306118426","heartbeat_back_addr":"172.21.15.46:6809/2306118426","heartbeat_front_addr":"172.21.15.46:6808/2306118426","state":["exists","up"]},{"osd":2,"uuid":"43f23619-ba49-4dda-b0a1-2a6578732b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6803","nonce":3493997451}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3493997451}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":3493997451}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6811","nonce":3493997451}]},"public_addr":"172.21.15.46:6803/3493997451","cluster_addr":"172.21.15.46:6802/3493997451","heartbeat_back_addr":"172.21.15.46:6810/3493997451","heartbeat_front_addr":"172.21.15.46:6811/3493997451","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-19T15:53:03.315 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-19T15:53:03.315 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-19T15:53:03.315 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-19T15:53:03.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:03.436+0000 7f100a19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:03.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:03.442+0000 7f100a19e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:03.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:03.439+0000 7f3ee639e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:03.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:03.439+0000 7f014e0f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:03.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:03.445+0000 7f014e0f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:03.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:03.446+0000 7f3ee639e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:03.462 INFO:teuthology.orchestra.run.smithi046.stdout:34359738373 2024-01-19T15:53:03.462 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-19T15:53:03.464 INFO:teuthology.orchestra.run.smithi046.stdout:34359738373 2024-01-19T15:53:03.464 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-19T15:53:03.465 INFO:teuthology.orchestra.run.smithi046.stdout:34359738373 2024-01-19T15:53:03.466 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-19T15:53:03.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:03.576+0000 7fb807e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:03.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:03.582+0000 7fb807e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:03.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:03.621+0000 7fa911b04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:03.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:03.622+0000 7fba8bf5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:03.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:03.628+0000 7fa911b04640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:03.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:03.628+0000 7fba8bf5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:03.772 INFO:teuthology.orchestra.run.smithi046.stdout:34359738373 2024-01-19T15:53:03.789 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2024-01-19T15:53:03.789 DEBUG:teuthology.parallel:result is None 2024-01-19T15:53:03.812 INFO:teuthology.orchestra.run.smithi046.stdout:34359738372 2024-01-19T15:53:03.813 INFO:teuthology.orchestra.run.smithi046.stdout:34359738373 2024-01-19T15:53:03.826 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2024-01-19T15:53:03.826 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2024-01-19T15:53:03.827 DEBUG:teuthology.parallel:result is None 2024-01-19T15:53:04.826 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-19T15:53:04.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:04.952+0000 7f2478a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:04.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:04.958+0000 7f2478a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:05.137 INFO:teuthology.orchestra.run.smithi046.stdout:34359738372 2024-01-19T15:53:05.149 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2024-01-19T15:53:06.150 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-19T15:53:06.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:06.276+0000 7f58e231e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:06.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:06.282+0000 7f58e231e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:06.459 INFO:teuthology.orchestra.run.smithi046.stdout:34359738373 2024-01-19T15:53:06.471 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2024-01-19T15:53:06.471 DEBUG:teuthology.parallel:result is None 2024-01-19T15:53:06.471 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-19T15:53:06.471 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-19T15:53:06.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:06.583+0000 7fd88cc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:06.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:06.588+0000 7fd88cc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:06.762 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:53:06.763 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-19T15:53:06.774 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-01-19T15:53:05.306362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80900,"kb_used_data":404,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034620,"statfs":{"total":289910292480,"available":289827450880,"internally_reserved":0,"allocated":413696,"data_stored":1248195,"data_compressed":59804,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.943775"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:53:03.367691+0000","last_change":"2024-01-19T15:53:02.430041+0000","last_active":"2024-01-19T15:53:03.367691+0000","last_peered":"2024-01-19T15:53:03.367691+0000","last_clean":"2024-01-19T15:53:03.367691+0000","last_became_active":"2024-01-19T15:53:02.428231+0000","last_became_peered":"2024-01-19T15:53:02.428231+0000","last_unstale":"2024-01-19T15:53:03.367691+0000","last_undegraded":"2024-01-19T15:53:03.367691+0000","last_fullsized":"2024-01-19T15:53:03.367691+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_clean_scrub_stamp":"2024-01-19T15:53:01.313832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:53:03.359133+0000","last_change":"2024-01-19T15:53:02.395520+0000","last_active":"2024-01-19T15:53:03.359133+0000","last_peered":"2024-01-19T15:53:03.359133+0000","last_clean":"2024-01-19T15:53:03.359133+0000","last_became_active":"2024-01-19T15:53:02.384946+0000","last_became_peered":"2024-01-19T15:53:02.384946+0000","last_unstale":"2024-01-19T15:53:03.359133+0000","last_undegraded":"2024-01-19T15:53:03.359133+0000","last_fullsized":"2024-01-19T15:53:03.359133+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_clean_scrub_stamp":"2024-01-19T15:53:01.313832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:53:03.366890+0000","last_change":"2024-01-19T15:53:02.407300+0000","last_active":"2024-01-19T15:53:03.366890+0000","last_peered":"2024-01-19T15:53:03.366890+0000","last_clean":"2024-01-19T15:53:03.366890+0000","last_became_active":"2024-01-19T15:53:02.405077+0000","last_became_peered":"2024-01-19T15:53:02.405077+0000","last_unstale":"2024-01-19T15:53:03.366890+0000","last_undegraded":"2024-01-19T15:53:03.366890+0000","last_fullsized":"2024-01-19T15:53:03.366890+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_clean_scrub_stamp":"2024-01-19T15:53:01.313832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:53:03.359661+0000","last_change":"2024-01-19T15:53:02.437576+0000","last_active":"2024-01-19T15:53:03.359661+0000","last_peered":"2024-01-19T15:53:03.359661+0000","last_clean":"2024-01-19T15:53:03.359661+0000","last_became_active":"2024-01-19T15:53:02.430974+0000","last_became_peered":"2024-01-19T15:53:02.430974+0000","last_unstale":"2024-01-19T15:53:03.359661+0000","last_undegraded":"2024-01-19T15:53:03.359661+0000","last_fullsized":"2024-01-19T15:53:03.359661+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_clean_scrub_stamp":"2024-01-19T15:53:01.313832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:53:03.361828+0000","last_change":"2024-01-19T15:53:02.425068+0000","last_active":"2024-01-19T15:53:03.361828+0000","last_peered":"2024-01-19T15:53:03.361828+0000","last_clean":"2024-01-19T15:53:03.361828+0000","last_became_active":"2024-01-19T15:53:02.422636+0000","last_became_peered":"2024-01-19T15:53:02.422636+0000","last_unstale":"2024-01-19T15:53:03.361828+0000","last_undegraded":"2024-01-19T15:53:03.361828+0000","last_fullsized":"2024-01-19T15:53:03.361828+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_clean_scrub_stamp":"2024-01-19T15:53:01.313832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:53:03.333764+0000","last_change":"2024-01-19T15:53:02.415076+0000","last_active":"2024-01-19T15:53:03.333764+0000","last_peered":"2024-01-19T15:53:03.333764+0000","last_clean":"2024-01-19T15:53:03.333764+0000","last_became_active":"2024-01-19T15:53:02.412852+0000","last_became_peered":"2024-01-19T15:53:02.412852+0000","last_unstale":"2024-01-19T15:53:03.333764+0000","last_undegraded":"2024-01-19T15:53:03.333764+0000","last_fullsized":"2024-01-19T15:53:03.333764+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_clean_scrub_stamp":"2024-01-19T15:53:01.313832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:53:03.335746+0000","last_change":"2024-01-19T15:53:02.409776+0000","last_active":"2024-01-19T15:53:03.335746+0000","last_peered":"2024-01-19T15:53:03.335746+0000","last_clean":"2024-01-19T15:53:03.335746+0000","last_became_active":"2024-01-19T15:53:02.407412+0000","last_became_peered":"2024-01-19T15:53:02.407412+0000","last_unstale":"2024-01-19T15:53:03.335746+0000","last_undegraded":"2024-01-19T15:53:03.335746+0000","last_fullsized":"2024-01-19T15:53:03.335746+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_clean_scrub_stamp":"2024-01-19T15:53:01.313832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:53:03.367765+0000","last_change":"2024-01-19T15:53:02.394121+0000","last_active":"2024-01-19T15:53:03.367765+0000","last_peered":"2024-01-19T15:53:03.367765+0000","last_clean":"2024-01-19T15:53:03.367765+0000","last_became_active":"2024-01-19T15:53:02.387989+0000","last_became_peered":"2024-01-19T15:53:02.387989+0000","last_unstale":"2024-01-19T15:53:03.367765+0000","last_undegraded":"2024-01-19T15:53:03.367765+0000","last_fullsized":"2024-01-19T15:53:03.367765+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_clean_scrub_stamp":"2024-01-19T15:53:01.313832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:53:03.360616+0000","last_change":"2024-01-19T15:53:01.401210+0000","last_active":"2024-01-19T15:53:03.360616+0000","last_peered":"2024-01-19T15:53:03.360616+0000","last_clean":"2024-01-19T15:53:03.360616+0000","last_became_active":"2024-01-19T15:53:01.375467+0000","last_became_peered":"2024-01-19T15:53:01.375467+0000","last_unstale":"2024-01-19T15:53:03.360616+0000","last_undegraded":"2024-01-19T15:53:03.360616+0000","last_fullsized":"2024-01-19T15:53:03.360616+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:53:00.311463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:53:00.311463+0000","last_clean_scrub_stamp":"2024-01-19T15:53:00.311463+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26916,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344924,"statfs":{"total":96636764160,"available":96609202176,"internally_reserved":0,"allocated":94208,"data_stored":21522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26988,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344852,"statfs":{"total":96636764160,"available":96609128448,"internally_reserved":0,"allocated":155648,"data_stored":611890,"data_compressed":29902,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":614783,"data_compressed":29902,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-19T15:53:06.775 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-19T15:53:06.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:06.887+0000 7f9ffc664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:06.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:06.892+0000 7f9ffc664640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:07.066 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:53:07.066 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-01-19T15:53:07.078 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-01-19T15:53:05.306362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80900,"kb_used_data":404,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034620,"statfs":{"total":289910292480,"available":289827450880,"internally_reserved":0,"allocated":413696,"data_stored":1248195,"data_compressed":59804,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.943775"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:53:03.367691+0000","last_change":"2024-01-19T15:53:02.430041+0000","last_active":"2024-01-19T15:53:03.367691+0000","last_peered":"2024-01-19T15:53:03.367691+0000","last_clean":"2024-01-19T15:53:03.367691+0000","last_became_active":"2024-01-19T15:53:02.428231+0000","last_became_peered":"2024-01-19T15:53:02.428231+0000","last_unstale":"2024-01-19T15:53:03.367691+0000","last_undegraded":"2024-01-19T15:53:03.367691+0000","last_fullsized":"2024-01-19T15:53:03.367691+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_clean_scrub_stamp":"2024-01-19T15:53:01.313832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:53:03.359133+0000","last_change":"2024-01-19T15:53:02.395520+0000","last_active":"2024-01-19T15:53:03.359133+0000","last_peered":"2024-01-19T15:53:03.359133+0000","last_clean":"2024-01-19T15:53:03.359133+0000","last_became_active":"2024-01-19T15:53:02.384946+0000","last_became_peered":"2024-01-19T15:53:02.384946+0000","last_unstale":"2024-01-19T15:53:03.359133+0000","last_undegraded":"2024-01-19T15:53:03.359133+0000","last_fullsized":"2024-01-19T15:53:03.359133+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_clean_scrub_stamp":"2024-01-19T15:53:01.313832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:53:03.366890+0000","last_change":"2024-01-19T15:53:02.407300+0000","last_active":"2024-01-19T15:53:03.366890+0000","last_peered":"2024-01-19T15:53:03.366890+0000","last_clean":"2024-01-19T15:53:03.366890+0000","last_became_active":"2024-01-19T15:53:02.405077+0000","last_became_peered":"2024-01-19T15:53:02.405077+0000","last_unstale":"2024-01-19T15:53:03.366890+0000","last_undegraded":"2024-01-19T15:53:03.366890+0000","last_fullsized":"2024-01-19T15:53:03.366890+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_clean_scrub_stamp":"2024-01-19T15:53:01.313832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:53:03.359661+0000","last_change":"2024-01-19T15:53:02.437576+0000","last_active":"2024-01-19T15:53:03.359661+0000","last_peered":"2024-01-19T15:53:03.359661+0000","last_clean":"2024-01-19T15:53:03.359661+0000","last_became_active":"2024-01-19T15:53:02.430974+0000","last_became_peered":"2024-01-19T15:53:02.430974+0000","last_unstale":"2024-01-19T15:53:03.359661+0000","last_undegraded":"2024-01-19T15:53:03.359661+0000","last_fullsized":"2024-01-19T15:53:03.359661+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_clean_scrub_stamp":"2024-01-19T15:53:01.313832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:53:03.361828+0000","last_change":"2024-01-19T15:53:02.425068+0000","last_active":"2024-01-19T15:53:03.361828+0000","last_peered":"2024-01-19T15:53:03.361828+0000","last_clean":"2024-01-19T15:53:03.361828+0000","last_became_active":"2024-01-19T15:53:02.422636+0000","last_became_peered":"2024-01-19T15:53:02.422636+0000","last_unstale":"2024-01-19T15:53:03.361828+0000","last_undegraded":"2024-01-19T15:53:03.361828+0000","last_fullsized":"2024-01-19T15:53:03.361828+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_clean_scrub_stamp":"2024-01-19T15:53:01.313832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:53:03.333764+0000","last_change":"2024-01-19T15:53:02.415076+0000","last_active":"2024-01-19T15:53:03.333764+0000","last_peered":"2024-01-19T15:53:03.333764+0000","last_clean":"2024-01-19T15:53:03.333764+0000","last_became_active":"2024-01-19T15:53:02.412852+0000","last_became_peered":"2024-01-19T15:53:02.412852+0000","last_unstale":"2024-01-19T15:53:03.333764+0000","last_undegraded":"2024-01-19T15:53:03.333764+0000","last_fullsized":"2024-01-19T15:53:03.333764+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_clean_scrub_stamp":"2024-01-19T15:53:01.313832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:53:03.335746+0000","last_change":"2024-01-19T15:53:02.409776+0000","last_active":"2024-01-19T15:53:03.335746+0000","last_peered":"2024-01-19T15:53:03.335746+0000","last_clean":"2024-01-19T15:53:03.335746+0000","last_became_active":"2024-01-19T15:53:02.407412+0000","last_became_peered":"2024-01-19T15:53:02.407412+0000","last_unstale":"2024-01-19T15:53:03.335746+0000","last_undegraded":"2024-01-19T15:53:03.335746+0000","last_fullsized":"2024-01-19T15:53:03.335746+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_clean_scrub_stamp":"2024-01-19T15:53:01.313832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:53:03.367765+0000","last_change":"2024-01-19T15:53:02.394121+0000","last_active":"2024-01-19T15:53:03.367765+0000","last_peered":"2024-01-19T15:53:03.367765+0000","last_clean":"2024-01-19T15:53:03.367765+0000","last_became_active":"2024-01-19T15:53:02.387989+0000","last_became_peered":"2024-01-19T15:53:02.387989+0000","last_unstale":"2024-01-19T15:53:03.367765+0000","last_undegraded":"2024-01-19T15:53:03.367765+0000","last_fullsized":"2024-01-19T15:53:03.367765+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:53:01.313832+0000","last_clean_scrub_stamp":"2024-01-19T15:53:01.313832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:53:03.360616+0000","last_change":"2024-01-19T15:53:01.401210+0000","last_active":"2024-01-19T15:53:03.360616+0000","last_peered":"2024-01-19T15:53:03.360616+0000","last_clean":"2024-01-19T15:53:03.360616+0000","last_became_active":"2024-01-19T15:53:01.375467+0000","last_became_peered":"2024-01-19T15:53:01.375467+0000","last_unstale":"2024-01-19T15:53:03.360616+0000","last_undegraded":"2024-01-19T15:53:03.360616+0000","last_fullsized":"2024-01-19T15:53:03.360616+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:53:00.311463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:53:00.311463+0000","last_clean_scrub_stamp":"2024-01-19T15:53:00.311463+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26916,"kb_used_data":92,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344924,"statfs":{"total":96636764160,"available":96609202176,"internally_reserved":0,"allocated":94208,"data_stored":21522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26988,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344852,"statfs":{"total":96636764160,"available":96609128448,"internally_reserved":0,"allocated":155648,"data_stored":611890,"data_compressed":29902,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":614783,"data_compressed":29902,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-19T15:53:07.079 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-19T15:53:07.079 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-19T15:53:07.079 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-19T15:53:07.079 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-19T15:53:07.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:07.191+0000 7f498f86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:53:07.195+0000 7f498f86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:07.407 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:53:07.407 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-19T15:53:07.419 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-19T15:53:07.420 INFO:teuthology.run_tasks:Running task cbt... 2024-01-19T15:53:07.436 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi046.front.sepia.ceph.com', 'osds': ['ubuntu@smithi046.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi046.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi046.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi046.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randread'], 'norandommap': True, 'op_size': [4096], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2], 'cmd_path': '/home/ubuntu/cephtest/fio/fio'}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-01-19T15:53:07.436 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-01-19T15:53:07.455 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-19T15:53:07.455 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-01-19T15:53:07.507 DEBUG:teuthology.orchestra.run.smithi046:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-01-19T15:53:07.560 INFO:teuthology.orchestra.run.smithi046.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-01-19T15:53:08.150 INFO:tasks.cbt:Installing collectl 2024-01-19T15:53:08.150 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-01-19T15:53:08.250 INFO:teuthology.orchestra.run.smithi046.stderr:--2024-01-19 15:53:08-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-01-19T15:53:08.406 INFO:teuthology.orchestra.run.smithi046.stderr:Resolving sourceforge.net (sourceforge.net)... 104.18.37.111, 172.64.150.145, 2606:4700:4400::ac40:9691, ... 2024-01-19T15:53:08.416 INFO:teuthology.orchestra.run.smithi046.stderr:Connecting to sourceforge.net (sourceforge.net)|104.18.37.111|:443... connected. 2024-01-19T15:53:08.640 INFO:teuthology.orchestra.run.smithi046.stderr:HTTP request sent, awaiting response... 302 Found 2024-01-19T15:53:08.640 INFO:teuthology.orchestra.run.smithi046.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlqprkZ9mCeVqkfmKwo4LSSK8SLiZPgQgd9P_U2ojPvuGf6rgmlooMfOu3yYVfJEYTSJUtshKTRrws-cZgceujwVU6dQ%3D%3D&use_mirror=gigenet&r= [following] 2024-01-19T15:53:08.640 INFO:teuthology.orchestra.run.smithi046.stderr:--2024-01-19 15:53:08-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlqprkZ9mCeVqkfmKwo4LSSK8SLiZPgQgd9P_U2ojPvuGf6rgmlooMfOu3yYVfJEYTSJUtshKTRrws-cZgceujwVU6dQ%3D%3D&use_mirror=gigenet&r= 2024-01-19T15:53:08.640 INFO:teuthology.orchestra.run.smithi046.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-01-19T15:53:08.715 INFO:teuthology.orchestra.run.smithi046.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-01-19T15:53:08.953 INFO:teuthology.orchestra.run.smithi046.stderr:HTTP request sent, awaiting response... 302 Found 2024-01-19T15:53:08.953 INFO:teuthology.orchestra.run.smithi046.stderr:Location: https://gigenet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2024-01-19T15:53:08.953 INFO:teuthology.orchestra.run.smithi046.stderr:--2024-01-19 15:53:08-- https://gigenet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2024-01-19T15:53:08.964 INFO:teuthology.orchestra.run.smithi046.stderr:Resolving gigenet.dl.sourceforge.net (gigenet.dl.sourceforge.net)... 69.65.16.142 2024-01-19T15:53:08.992 INFO:teuthology.orchestra.run.smithi046.stderr:Connecting to gigenet.dl.sourceforge.net (gigenet.dl.sourceforge.net)|69.65.16.142|:443... connected. 2024-01-19T15:53:09.407 INFO:teuthology.orchestra.run.smithi046.stderr:HTTP request sent, awaiting response... 200 OK 2024-01-19T15:53:09.407 INFO:teuthology.orchestra.run.smithi046.stderr:Length: 620693 (606K) [application/x-gzip] 2024-01-19T15:53:09.407 INFO:teuthology.orchestra.run.smithi046.stderr:Saving to: ‘collectl.tar.gz’ 2024-01-19T15:53:09.407 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-01-19T15:53:09.464 INFO:teuthology.orchestra.run.smithi046.stderr: 0K .......... .......... .......... .......... .......... 8% 873K 1s 2024-01-19T15:53:09.519 INFO:teuthology.orchestra.run.smithi046.stderr: 50K .......... .......... .......... .......... .......... 16% 907K 1s 2024-01-19T15:53:09.522 INFO:teuthology.orchestra.run.smithi046.stderr: 100K .......... .......... .......... .......... .......... 24% 13.3M 0s 2024-01-19T15:53:09.550 INFO:teuthology.orchestra.run.smithi046.stderr: 150K .......... .......... .......... .......... .......... 32% 1.79M 0s 2024-01-19T15:53:09.574 INFO:teuthology.orchestra.run.smithi046.stderr: 200K .......... .......... .......... .......... .......... 41% 2.04M 0s 2024-01-19T15:53:09.578 INFO:teuthology.orchestra.run.smithi046.stderr: 250K .......... .......... .......... .......... .......... 49% 12.6M 0s 2024-01-19T15:53:09.601 INFO:teuthology.orchestra.run.smithi046.stderr: 300K .......... .......... .......... .......... .......... 57% 2.07M 0s 2024-01-19T15:53:09.606 INFO:teuthology.orchestra.run.smithi046.stderr: 350K .......... .......... .......... .......... .......... 65% 13.2M 0s 2024-01-19T15:53:09.606 INFO:teuthology.orchestra.run.smithi046.stderr: 400K .......... .......... .......... .......... .......... 74% 67.1M 0s 2024-01-19T15:53:09.629 INFO:teuthology.orchestra.run.smithi046.stderr: 450K .......... .......... .......... .......... .......... 82% 2.10M 0s 2024-01-19T15:53:09.633 INFO:teuthology.orchestra.run.smithi046.stderr: 500K .......... .......... .......... .......... .......... 90% 14.6M 0s 2024-01-19T15:53:09.633 INFO:teuthology.orchestra.run.smithi046.stderr: 550K .......... .......... .......... .......... .......... 98% 96.8M 0s 2024-01-19T15:53:09.633 INFO:teuthology.orchestra.run.smithi046.stderr: 600K ...... 100% 11.4T=0.2s 2024-01-19T15:53:09.633 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-01-19T15:53:09.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19 15:53:09 (2.62 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-01-19T15:53:09.633 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-01-19T15:53:09.673 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/ 2024-01-19T15:53:09.674 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/collectl.conf 2024-01-19T15:53:09.674 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/ 2024-01-19T15:53:09.674 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/SlowProc.html 2024-01-19T15:53:09.674 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-01-19T15:53:09.674 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/WhySummary.html 2024-01-19T15:53:09.674 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Socket.html 2024-01-19T15:53:09.674 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/colmux.html 2024-01-19T15:53:09.674 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Tutorial.html 2024-01-19T15:53:09.674 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Gexpr.html 2024-01-19T15:53:09.674 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Import.html 2024-01-19T15:53:09.674 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/style.css 2024-01-19T15:53:09.675 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-01-19T15:53:09.675 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-01-19T15:53:09.675 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/FileNaming.html 2024-01-19T15:53:09.675 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/TheMath.html 2024-01-19T15:53:09.675 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Environmental.html 2024-01-19T15:53:09.675 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Process.html 2024-01-19T15:53:09.675 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-01-19T15:53:09.675 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/HiResTime.html 2024-01-19T15:53:09.675 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-01-19T15:53:09.675 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/slub.jpg 2024-01-19T15:53:09.675 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Lustre.html 2024-01-19T15:53:09.675 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Documentation.html 2024-01-19T15:53:09.676 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Inodes.html 2024-01-19T15:53:09.676 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Graphite.html 2024-01-19T15:53:09.676 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/InputFiles.html 2024-01-19T15:53:09.676 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Export.html 2024-01-19T15:53:09.676 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Interrupts.html 2024-01-19T15:53:09.676 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Startup.html 2024-01-19T15:53:09.676 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Performance.html 2024-01-19T15:53:09.676 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Messages.html 2024-01-19T15:53:09.676 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Logging.html 2024-01-19T15:53:09.676 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Data-detail.html 2024-01-19T15:53:09.676 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/CPUs.html 2024-01-19T15:53:09.676 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Matrix.html 2024-01-19T15:53:09.676 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Data-brief.html 2024-01-19T15:53:09.676 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-01-19T15:53:09.677 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Misc.html 2024-01-19T15:53:09.677 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Examples.html 2024-01-19T15:53:09.677 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/OpenStack.html 2024-01-19T15:53:09.677 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-01-19T15:53:09.677 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-01-19T15:53:09.677 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-01-19T15:53:09.677 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-01-19T15:53:09.677 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Data.html 2024-01-19T15:53:09.677 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Exceptions.html 2024-01-19T15:53:09.677 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-01-19T15:53:09.677 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-01-19T15:53:09.677 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-01-19T15:53:09.677 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-01-19T15:53:09.677 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Playback.html 2024-01-19T15:53:09.678 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Hello.html 2024-01-19T15:53:09.678 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Sockets.html 2024-01-19T15:53:09.678 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Network.html 2024-01-19T15:53:09.678 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Infiniband.html 2024-01-19T15:53:09.678 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Memory.html 2024-01-19T15:53:09.678 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-01-19T15:53:09.678 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/Disks.html 2024-01-19T15:53:09.678 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-01-19T15:53:09.678 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/service/ 2024-01-19T15:53:09.678 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/service/collectl.service 2024-01-19T15:53:09.678 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/initd/ 2024-01-19T15:53:09.678 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/initd/collectl-suse 2024-01-19T15:53:09.678 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/initd/collectl-debian 2024-01-19T15:53:09.678 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/initd/collectl-generic 2024-01-19T15:53:09.678 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/initd/collectl 2024-01-19T15:53:09.679 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/misc.ph 2024-01-19T15:53:09.679 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/gexpr.ph 2024-01-19T15:53:09.679 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/statsd.ph 2024-01-19T15:53:09.679 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/envrules.std 2024-01-19T15:53:09.679 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/hello.ph 2024-01-19T15:53:09.679 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/vmsum.ph 2024-01-19T15:53:09.679 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/proctree.ph 2024-01-19T15:53:09.679 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/README-WINDOWS 2024-01-19T15:53:09.679 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/formatit.ph 2024-01-19T15:53:09.679 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/colmux 2024-01-19T15:53:09.679 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/INSTALL 2024-01-19T15:53:09.679 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/graphite.ph 2024-01-19T15:53:09.679 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/RELEASE-collectl 2024-01-19T15:53:09.680 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/vmstat.ph 2024-01-19T15:53:09.680 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/README 2024-01-19T15:53:09.680 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/man1/ 2024-01-19T15:53:09.680 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/man1/collectl.1 2024-01-19T15:53:09.680 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/man1/colmux.1 2024-01-19T15:53:09.680 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/lexpr.ph 2024-01-19T15:53:09.680 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/client.pl 2024-01-19T15:53:09.680 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/collectl 2024-01-19T15:53:09.681 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/COPYING 2024-01-19T15:53:09.681 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/vnet.ph 2024-01-19T15:53:09.681 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/GPL 2024-01-19T15:53:09.681 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/UNINSTALL 2024-01-19T15:53:09.681 INFO:teuthology.orchestra.run.smithi046.stdout:collectl-4.3.1/ARTISTIC 2024-01-19T15:53:09.732 INFO:teuthology.orchestra.run.smithi046.stderr:./INSTALL: line 90: chkconfig: command not found 2024-01-19T15:53:09.734 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-01-19T15:53:10.303 INFO:teuthology.orchestra.run.smithi046.stdout:Last metadata expiration check: 0:04:10 ago on Fri 19 Jan 2024 03:49:00 PM UTC. 2024-01-19T15:53:10.427 INFO:teuthology.orchestra.run.smithi046.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-01-19T15:53:10.430 INFO:teuthology.orchestra.run.smithi046.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-01-19T15:53:10.489 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-01-19T15:53:10.490 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:53:10.490 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-01-19T15:53:10.490 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:53:10.490 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-01-19T15:53:10.490 INFO:teuthology.orchestra.run.smithi046.stdout: librbd-devel x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 23 k 2024-01-19T15:53:10.490 INFO:teuthology.orchestra.run.smithi046.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-01-19T15:53:10.490 INFO:teuthology.orchestra.run.smithi046.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-01-19T15:53:10.491 INFO:teuthology.orchestra.run.smithi046.stdout: perf x86_64 5.14.0-409.el9 CentOS-AppStream 8.0 M 2024-01-19T15:53:10.491 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-01-19T15:53:10.491 INFO:teuthology.orchestra.run.smithi046.stdout: libradospp-devel x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 30 k 2024-01-19T15:53:10.491 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:53:10.491 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-01-19T15:53:10.491 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-01-19T15:53:10.491 INFO:teuthology.orchestra.run.smithi046.stdout:Install 5 Packages 2024-01-19T15:53:10.491 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:53:10.492 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 8.1 M 2024-01-19T15:53:10.492 INFO:teuthology.orchestra.run.smithi046.stdout:Installed size: 11 M 2024-01-19T15:53:10.492 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-01-19T15:53:11.102 INFO:teuthology.orchestra.run.smithi046.stdout:(1/5): librbd-devel-19.0.0-767.gbdd6070c.el9.x8 93 kB/s | 23 kB 00:00 2024-01-19T15:53:11.127 INFO:teuthology.orchestra.run.smithi046.stdout:(2/5): libradospp-devel-19.0.0-767.gbdd6070c.el 111 kB/s | 30 kB 00:00 2024-01-19T15:53:11.361 INFO:teuthology.orchestra.run.smithi046.stdout:(3/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 56 kB/s | 13 kB 00:00 2024-01-19T15:53:11.453 INFO:teuthology.orchestra.run.smithi046.stdout:(4/5): perf-5.14.0-409.el9.x86_64.rpm 13 MB/s | 8.0 MB 00:00 2024-01-19T15:53:11.478 INFO:teuthology.orchestra.run.smithi046.stdout:(5/5): pdsh-2.34-7.el9.x86_64.rpm 316 kB/s | 118 kB 00:00 2024-01-19T15:53:11.481 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:53:11.482 INFO:teuthology.orchestra.run.smithi046.stdout:Total 8.2 MB/s | 8.1 MB 00:00 2024-01-19T15:53:11.482 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-01-19T15:53:11.504 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-01-19T15:53:11.504 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-01-19T15:53:11.668 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-01-19T15:53:11.668 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-01-19T15:53:12.054 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-01-19T15:53:12.123 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-01-19T15:53:12.152 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-01-19T15:53:12.184 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libradospp-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 3/5 2024-01-19T15:53:12.526 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librbd-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 4/5 2024-01-19T15:53:12.667 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : perf-5.14.0-409.el9.x86_64 5/5 2024-01-19T15:53:13.571 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: perf-5.14.0-409.el9.x86_64 5/5 2024-01-19T15:53:13.572 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libradospp-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 1/5 2024-01-19T15:53:13.572 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 2/5 2024-01-19T15:53:13.572 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : perf-5.14.0-409.el9.x86_64 3/5 2024-01-19T15:53:13.572 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-01-19T15:53:13.922 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-01-19T15:53:13.923 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:53:13.923 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-01-19T15:53:13.923 INFO:teuthology.orchestra.run.smithi046.stdout: libradospp-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:53:13.923 INFO:teuthology.orchestra.run.smithi046.stdout: librbd-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:53:13.923 INFO:teuthology.orchestra.run.smithi046.stdout: pdsh-2.34-7.el9.x86_64 2024-01-19T15:53:13.923 INFO:teuthology.orchestra.run.smithi046.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-01-19T15:53:13.923 INFO:teuthology.orchestra.run.smithi046.stdout: perf-5.14.0-409.el9.x86_64 2024-01-19T15:53:13.923 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:53:13.923 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-01-19T15:53:14.279 INFO:tasks.cbt:benchmark: librbdfio 2024-01-19T15:53:14.279 DEBUG:teuthology.orchestra.run.smithi046:> git clone -b master https://github.com/axboe/fio.git /home/ubuntu/cephtest/fio 2024-01-19T15:53:14.294 INFO:teuthology.orchestra.run.smithi046.stderr:Cloning into '/home/ubuntu/cephtest/fio'... 2024-01-19T15:53:16.699 DEBUG:teuthology.orchestra.run.smithi046:> cd /home/ubuntu/cephtest/fio && ./configure && make 2024-01-19T15:53:17.283 INFO:teuthology.orchestra.run.smithi046.stdout:Operating system Linux 2024-01-19T15:53:17.283 INFO:teuthology.orchestra.run.smithi046.stdout:CPU x86_64 2024-01-19T15:53:17.283 INFO:teuthology.orchestra.run.smithi046.stdout:Big endian no 2024-01-19T15:53:17.283 INFO:teuthology.orchestra.run.smithi046.stdout:Compiler gcc 2024-01-19T15:53:17.283 INFO:teuthology.orchestra.run.smithi046.stdout:Cross compile no 2024-01-19T15:53:17.283 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:53:17.284 INFO:teuthology.orchestra.run.smithi046.stdout:Static build no 2024-01-19T15:53:17.415 INFO:teuthology.orchestra.run.smithi046.stdout:Wordsize 64 2024-01-19T15:53:17.422 INFO:teuthology.orchestra.run.smithi046.stdout:zlib no 2024-01-19T15:53:17.459 INFO:teuthology.orchestra.run.smithi046.stdout:fcntl(F_FULLFSYNC) no 2024-01-19T15:53:17.473 INFO:teuthology.orchestra.run.smithi046.stdout:Linux AIO support no 2024-01-19T15:53:17.473 INFO:teuthology.orchestra.run.smithi046.stdout:Linux AIO support rw flags no 2024-01-19T15:53:17.537 INFO:teuthology.orchestra.run.smithi046.stdout:POSIX AIO support yes 2024-01-19T15:53:17.538 INFO:teuthology.orchestra.run.smithi046.stdout:POSIX AIO support needs -lrt no 2024-01-19T15:53:17.567 INFO:teuthology.orchestra.run.smithi046.stdout:POSIX AIO fsync yes 2024-01-19T15:53:17.597 INFO:teuthology.orchestra.run.smithi046.stdout:POSIX pshared support yes 2024-01-19T15:53:17.647 INFO:teuthology.orchestra.run.smithi046.stdout:pthread_condattr_setclock() yes 2024-01-19T15:53:17.685 INFO:teuthology.orchestra.run.smithi046.stdout:pthread_sigmask() yes 2024-01-19T15:53:17.721 INFO:teuthology.orchestra.run.smithi046.stdout:pthread_getaffinity_np() yes 2024-01-19T15:53:17.730 INFO:teuthology.orchestra.run.smithi046.stdout:Solaris AIO support no 2024-01-19T15:53:17.781 INFO:teuthology.orchestra.run.smithi046.stdout:__sync_fetch_and_add yes 2024-01-19T15:53:17.820 INFO:teuthology.orchestra.run.smithi046.stdout:__sync_synchronize yes 2024-01-19T15:53:17.848 INFO:teuthology.orchestra.run.smithi046.stdout:__sync_val_compare_and_swap yes 2024-01-19T15:53:17.855 INFO:teuthology.orchestra.run.smithi046.stdout:libverbs no 2024-01-19T15:53:17.880 INFO:teuthology.orchestra.run.smithi046.stdout:rdmacm no 2024-01-19T15:53:17.887 INFO:teuthology.orchestra.run.smithi046.stdout:librpma no 2024-01-19T15:53:17.897 INFO:teuthology.orchestra.run.smithi046.stdout:libprotobuf_c no 2024-01-19T15:53:17.927 INFO:teuthology.orchestra.run.smithi046.stdout:asprintf() yes 2024-01-19T15:53:17.957 INFO:teuthology.orchestra.run.smithi046.stdout:vasprintf() yes 2024-01-19T15:53:17.988 INFO:teuthology.orchestra.run.smithi046.stdout:Linux fallocate yes 2024-01-19T15:53:18.019 INFO:teuthology.orchestra.run.smithi046.stdout:POSIX fadvise yes 2024-01-19T15:53:18.050 INFO:teuthology.orchestra.run.smithi046.stdout:POSIX fallocate yes 2024-01-19T15:53:18.080 INFO:teuthology.orchestra.run.smithi046.stdout:sched_setaffinity(3 arg) yes 2024-01-19T15:53:18.080 INFO:teuthology.orchestra.run.smithi046.stdout:sched_setaffinity(2 arg) no 2024-01-19T15:53:18.110 INFO:teuthology.orchestra.run.smithi046.stdout:clock_gettime yes 2024-01-19T15:53:18.141 INFO:teuthology.orchestra.run.smithi046.stdout:CLOCK_MONOTONIC yes 2024-01-19T15:53:18.181 INFO:teuthology.orchestra.run.smithi046.stdout:clockid_t yes 2024-01-19T15:53:18.221 INFO:teuthology.orchestra.run.smithi046.stdout:gettimeofday yes 2024-01-19T15:53:18.253 INFO:teuthology.orchestra.run.smithi046.stdout:fdatasync yes 2024-01-19T15:53:18.282 INFO:teuthology.orchestra.run.smithi046.stdout:pipe() yes 2024-01-19T15:53:18.313 INFO:teuthology.orchestra.run.smithi046.stdout:pipe2() yes 2024-01-19T15:53:18.344 INFO:teuthology.orchestra.run.smithi046.stdout:pread() yes 2024-01-19T15:53:18.431 INFO:teuthology.orchestra.run.smithi046.stdout:sync_file_range yes 2024-01-19T15:53:18.438 INFO:teuthology.orchestra.run.smithi046.stdout:ASharedMemory_create no 2024-01-19T15:53:18.450 INFO:teuthology.orchestra.run.smithi046.stdout:EXT4 move extent yes 2024-01-19T15:53:18.481 INFO:teuthology.orchestra.run.smithi046.stdout:Linux splice(2) yes 2024-01-19T15:53:18.488 INFO:teuthology.orchestra.run.smithi046.stdout:libnuma no 2024-01-19T15:53:18.517 INFO:teuthology.orchestra.run.smithi046.stdout:strsep yes 2024-01-19T15:53:18.546 INFO:teuthology.orchestra.run.smithi046.stdout:strcasestr yes 2024-01-19T15:53:18.560 INFO:teuthology.orchestra.run.smithi046.stdout:strlcat no 2024-01-19T15:53:18.593 INFO:teuthology.orchestra.run.smithi046.stdout:getopt_long_only() yes 2024-01-19T15:53:18.669 INFO:teuthology.orchestra.run.smithi046.stdout:inet_aton yes 2024-01-19T15:53:18.699 INFO:teuthology.orchestra.run.smithi046.stdout:socklen_t yes 2024-01-19T15:53:18.731 INFO:teuthology.orchestra.run.smithi046.stdout:__thread yes 2024-01-19T15:53:18.761 INFO:teuthology.orchestra.run.smithi046.stdout:RUSAGE_THREAD yes 2024-01-19T15:53:18.790 INFO:teuthology.orchestra.run.smithi046.stdout:SCHED_IDLE yes 2024-01-19T15:53:18.824 INFO:teuthology.orchestra.run.smithi046.stdout:TCP_NODELAY yes 2024-01-19T15:53:18.857 INFO:teuthology.orchestra.run.smithi046.stdout:Net engine window_size yes 2024-01-19T15:53:18.890 INFO:teuthology.orchestra.run.smithi046.stdout:TCP_MAXSEG yes 2024-01-19T15:53:18.919 INFO:teuthology.orchestra.run.smithi046.stdout:RLIMIT_MEMLOCK yes 2024-01-19T15:53:18.951 INFO:teuthology.orchestra.run.smithi046.stdout:pwritev/preadv yes 2024-01-19T15:53:18.983 INFO:teuthology.orchestra.run.smithi046.stdout:pwritev2/preadv2 yes 2024-01-19T15:53:19.018 INFO:teuthology.orchestra.run.smithi046.stdout:IPv6 helpers yes 2024-01-19T15:53:19.033 INFO:teuthology.orchestra.run.smithi046.stdout:http engine no 2024-01-19T15:53:19.128 INFO:teuthology.orchestra.run.smithi046.stdout:Rados engine yes 2024-01-19T15:53:19.232 INFO:teuthology.orchestra.run.smithi046.stdout:Rados Block Device engine yes 2024-01-19T15:53:19.341 INFO:teuthology.orchestra.run.smithi046.stdout:rbd_poll yes 2024-01-19T15:53:19.446 INFO:teuthology.orchestra.run.smithi046.stdout:rbd_invalidate_cache yes 2024-01-19T15:53:19.476 INFO:teuthology.orchestra.run.smithi046.stdout:setvbuf yes 2024-01-19T15:53:19.483 INFO:teuthology.orchestra.run.smithi046.stdout:Gluster API engine no 2024-01-19T15:53:19.499 INFO:teuthology.orchestra.run.smithi046.stdout:s390_z196_facilities no 2024-01-19T15:53:19.499 INFO:teuthology.orchestra.run.smithi046.stdout:HDFS engine no 2024-01-19T15:53:19.568 INFO:teuthology.orchestra.run.smithi046.stdout:MTD yes 2024-01-19T15:53:19.575 INFO:teuthology.orchestra.run.smithi046.stdout:libpmem no 2024-01-19T15:53:19.575 INFO:teuthology.orchestra.run.smithi046.stdout:libpmem1_5 no 2024-01-19T15:53:19.582 INFO:teuthology.orchestra.run.smithi046.stdout:libpmem2 no 2024-01-19T15:53:19.583 INFO:teuthology.orchestra.run.smithi046.stdout:PMDK dev-dax engine no 2024-01-19T15:53:19.583 INFO:teuthology.orchestra.run.smithi046.stdout:PMDK libpmem engine no 2024-01-19T15:53:19.589 INFO:teuthology.orchestra.run.smithi046.stdout:DDN's Infinite Memory Engine no 2024-01-19T15:53:19.589 INFO:teuthology.orchestra.run.smithi046.stdout:iscsi engine no 2024-01-19T15:53:19.590 INFO:teuthology.orchestra.run.smithi046.stdout:NBD engine no 2024-01-19T15:53:19.598 INFO:teuthology.orchestra.run.smithi046.stdout:DAOS File System (dfs) Engine no 2024-01-19T15:53:19.607 INFO:teuthology.orchestra.run.smithi046.stdout:NFS engine no 2024-01-19T15:53:19.607 INFO:teuthology.orchestra.run.smithi046.stdout:lex/yacc for arithmetic no 2024-01-19T15:53:19.644 INFO:teuthology.orchestra.run.smithi046.stdout:getmntent yes 2024-01-19T15:53:19.677 INFO:teuthology.orchestra.run.smithi046.stdout:getmntinfo no 2024-01-19T15:53:19.733 INFO:teuthology.orchestra.run.smithi046.stdout:Static Assert yes 2024-01-19T15:53:19.760 INFO:teuthology.orchestra.run.smithi046.stdout:bool yes 2024-01-19T15:53:19.793 INFO:teuthology.orchestra.run.smithi046.stdout:strndup yes 2024-01-19T15:53:19.800 INFO:teuthology.orchestra.run.smithi046.stdout:Valgrind headers no 2024-01-19T15:53:19.835 INFO:teuthology.orchestra.run.smithi046.stdout:Zoned block device support yes 2024-01-19T15:53:19.863 INFO:teuthology.orchestra.run.smithi046.stdout:Zoned block device capacity yes 2024-01-19T15:53:19.870 INFO:teuthology.orchestra.run.smithi046.stdout:libzbc engine no 2024-01-19T15:53:19.902 INFO:teuthology.orchestra.run.smithi046.stdout:NVMe uring command support yes 2024-01-19T15:53:19.916 INFO:teuthology.orchestra.run.smithi046.stdout:xnvme engine no 2024-01-19T15:53:19.923 INFO:teuthology.orchestra.run.smithi046.stdout:isal no 2024-01-19T15:53:19.938 INFO:teuthology.orchestra.run.smithi046.stdout:libblkio engine no 2024-01-19T15:53:19.938 INFO:teuthology.orchestra.run.smithi046.stdout:march_armv8_a_crc_crypto no 2024-01-19T15:53:19.938 INFO:teuthology.orchestra.run.smithi046.stdout:cuda no 2024-01-19T15:53:19.938 INFO:teuthology.orchestra.run.smithi046.stdout:libcufile no 2024-01-19T15:53:19.965 INFO:teuthology.orchestra.run.smithi046.stdout:Build march=native yes 2024-01-19T15:53:19.974 INFO:teuthology.orchestra.run.smithi046.stdout:CUnit no 2024-01-19T15:53:20.001 INFO:teuthology.orchestra.run.smithi046.stdout:__kernel_rwf_t yes 2024-01-19T15:53:20.028 INFO:teuthology.orchestra.run.smithi046.stdout:-Wimplicit-fallthrough=2 yes 2024-01-19T15:53:20.058 INFO:teuthology.orchestra.run.smithi046.stdout:-Wno-stringop-truncation yes 2024-01-19T15:53:20.089 INFO:teuthology.orchestra.run.smithi046.stdout:MADV_HUGEPAGE yes 2024-01-19T15:53:20.119 INFO:teuthology.orchestra.run.smithi046.stdout:gettid yes 2024-01-19T15:53:20.152 INFO:teuthology.orchestra.run.smithi046.stdout:statx(2)/libc yes 2024-01-19T15:53:20.185 INFO:teuthology.orchestra.run.smithi046.stdout:statx(2)/syscall yes 2024-01-19T15:53:20.187 INFO:teuthology.orchestra.run.smithi046.stdout:Windows PDB generation no 2024-01-19T15:53:20.217 INFO:teuthology.orchestra.run.smithi046.stdout:timerfd_create yes 2024-01-19T15:53:20.219 INFO:teuthology.orchestra.run.smithi046.stdout:Consider installing zlib1g-dev (zlib-devel) as some fio features depend on it. 2024-01-19T15:53:20.219 INFO:teuthology.orchestra.run.smithi046.stdout:Lib-based ioengines dynamic no 2024-01-19T15:53:20.281 INFO:teuthology.orchestra.run.smithi046.stdout:TCMalloc support yes 2024-01-19T15:53:20.282 INFO:teuthology.orchestra.run.smithi046.stdout:seed_buckets 4 2024-01-19T15:53:20.361 INFO:teuthology.orchestra.run.smithi046.stderr:FIO_VERSION = fio-3.36-42-gaa84 2024-01-19T15:53:20.394 INFO:teuthology.orchestra.run.smithi046.stdout: CC crc/crc16.o 2024-01-19T15:53:20.483 INFO:teuthology.orchestra.run.smithi046.stdout: CC crc/crc32.o 2024-01-19T15:53:20.519 INFO:teuthology.orchestra.run.smithi046.stdout: CC crc/crc32c-arm64.o 2024-01-19T15:53:20.591 INFO:teuthology.orchestra.run.smithi046.stdout: CC crc/crc32c-intel.o 2024-01-19T15:53:20.633 INFO:teuthology.orchestra.run.smithi046.stdout: CC crc/crc32c.o 2024-01-19T15:53:20.671 INFO:teuthology.orchestra.run.smithi046.stdout: CC crc/crc64.o 2024-01-19T15:53:20.710 INFO:teuthology.orchestra.run.smithi046.stdout: CC crc/crc7.o 2024-01-19T15:53:20.742 INFO:teuthology.orchestra.run.smithi046.stdout: CC crc/crct10dif_common.o 2024-01-19T15:53:20.775 INFO:teuthology.orchestra.run.smithi046.stdout: CC crc/fnv.o 2024-01-19T15:53:20.814 INFO:teuthology.orchestra.run.smithi046.stdout: CC crc/md5.o 2024-01-19T15:53:20.919 INFO:teuthology.orchestra.run.smithi046.stdout: CC crc/murmur3.o 2024-01-19T15:53:20.960 INFO:teuthology.orchestra.run.smithi046.stdout: CC crc/sha1.o 2024-01-19T15:53:21.144 INFO:teuthology.orchestra.run.smithi046.stdout: CC crc/sha256.o 2024-01-19T15:53:21.442 INFO:teuthology.orchestra.run.smithi046.stdout: CC crc/sha3.o 2024-01-19T15:53:21.587 INFO:teuthology.orchestra.run.smithi046.stdout: CC crc/sha512.o 2024-01-19T15:53:21.698 INFO:teuthology.orchestra.run.smithi046.stdout: CC crc/test.o 2024-01-19T15:53:21.864 INFO:teuthology.orchestra.run.smithi046.stdout: CC crc/xxhash.o 2024-01-19T15:53:21.961 INFO:teuthology.orchestra.run.smithi046.stdout: CC lib/axmap.o 2024-01-19T15:53:22.087 INFO:teuthology.orchestra.run.smithi046.stdout: CC lib/bloom.o 2024-01-19T15:53:22.170 INFO:teuthology.orchestra.run.smithi046.stdout: CC lib/flist_sort.o 2024-01-19T15:53:22.242 INFO:teuthology.orchestra.run.smithi046.stdout: CC lib/gauss.o 2024-01-19T15:53:22.339 INFO:teuthology.orchestra.run.smithi046.stdout: CC lib/getrusage.o 2024-01-19T15:53:22.373 INFO:teuthology.orchestra.run.smithi046.stdout: CC lib/hweight.o 2024-01-19T15:53:22.410 INFO:teuthology.orchestra.run.smithi046.stdout: CC lib/ieee754.o 2024-01-19T15:53:22.459 INFO:teuthology.orchestra.run.smithi046.stdout: CC lib/lfsr.o 2024-01-19T15:53:22.562 INFO:teuthology.orchestra.run.smithi046.stdout: CC lib/memalign.o 2024-01-19T15:53:22.605 INFO:teuthology.orchestra.run.smithi046.stdout: CC lib/memcpy.o 2024-01-19T15:53:22.752 INFO:teuthology.orchestra.run.smithi046.stdout: CC lib/mountcheck.o 2024-01-19T15:53:22.791 INFO:teuthology.orchestra.run.smithi046.stdout: CC lib/num2str.o 2024-01-19T15:53:22.856 INFO:teuthology.orchestra.run.smithi046.stdout: CC lib/output_buffer.o 2024-01-19T15:53:22.903 INFO:teuthology.orchestra.run.smithi046.stdout: CC lib/pattern.o 2024-01-19T15:53:23.050 INFO:teuthology.orchestra.run.smithi046.stdout: CC lib/prio_tree.o 2024-01-19T15:53:23.171 INFO:teuthology.orchestra.run.smithi046.stdout: CC lib/rand.o 2024-01-19T15:53:23.361 INFO:teuthology.orchestra.run.smithi046.stdout: CC lib/rbtree.o 2024-01-19T15:53:23.488 INFO:teuthology.orchestra.run.smithi046.stdout: CC lib/strntol.o 2024-01-19T15:53:23.534 INFO:teuthology.orchestra.run.smithi046.stdout: CC lib/zipf.o 2024-01-19T15:53:23.655 INFO:teuthology.orchestra.run.smithi046.stdout: CC gettime.o 2024-01-19T15:53:23.952 INFO:teuthology.orchestra.run.smithi046.stdout: CC ioengines.o 2024-01-19T15:53:24.231 INFO:teuthology.orchestra.run.smithi046.stdout: CC init.o 2024-01-19T15:53:25.006 INFO:teuthology.orchestra.run.smithi046.stdout: CC stat.o 2024-01-19T15:53:26.957 INFO:teuthology.orchestra.run.smithi046.stdout: CC log.o 2024-01-19T15:53:27.079 INFO:teuthology.orchestra.run.smithi046.stdout: CC time.o 2024-01-19T15:53:27.203 INFO:teuthology.orchestra.run.smithi046.stdout: CC filesetup.o 2024-01-19T15:53:27.920 INFO:teuthology.orchestra.run.smithi046.stdout: CC eta.o 2024-01-19T15:53:28.251 INFO:teuthology.orchestra.run.smithi046.stdout: CC verify.o 2024-01-19T15:53:28.946 INFO:teuthology.orchestra.run.smithi046.stdout: CC memory.o 2024-01-19T15:53:29.117 INFO:teuthology.orchestra.run.smithi046.stdout: CC io_u.o 2024-01-19T15:53:29.900 INFO:teuthology.orchestra.run.smithi046.stdout: CC parse.o 2024-01-19T15:53:30.355 INFO:teuthology.orchestra.run.smithi046.stdout: CC fio_sem.o 2024-01-19T15:53:30.460 INFO:teuthology.orchestra.run.smithi046.stdout: CC rwlock.o 2024-01-19T15:53:30.541 INFO:teuthology.orchestra.run.smithi046.stdout: CC pshared.o 2024-01-19T15:53:30.605 INFO:teuthology.orchestra.run.smithi046.stdout: CC options.o 2024-01-19T15:53:31.410 INFO:teuthology.orchestra.run.smithi046.stdout: CC smalloc.o 2024-01-19T15:53:31.595 INFO:teuthology.orchestra.run.smithi046.stdout: CC filehash.o 2024-01-19T15:53:31.746 INFO:teuthology.orchestra.run.smithi046.stdout: CC profile.o 2024-01-19T15:53:31.863 INFO:teuthology.orchestra.run.smithi046.stdout: CC debug.o 2024-01-19T15:53:31.902 INFO:teuthology.orchestra.run.smithi046.stdout: CC engines/cpu.o 2024-01-19T15:53:32.039 INFO:teuthology.orchestra.run.smithi046.stdout: CC engines/mmap.o 2024-01-19T15:53:32.211 INFO:teuthology.orchestra.run.smithi046.stdout: CC engines/sync.o 2024-01-19T15:53:32.440 INFO:teuthology.orchestra.run.smithi046.stdout: CC engines/null.o 2024-01-19T15:53:32.558 INFO:teuthology.orchestra.run.smithi046.stdout: CC engines/net.o 2024-01-19T15:53:32.967 INFO:teuthology.orchestra.run.smithi046.stdout: CC engines/ftruncate.o 2024-01-19T15:53:33.063 INFO:teuthology.orchestra.run.smithi046.stdout: CC engines/fileoperations.o 2024-01-19T15:53:33.204 INFO:teuthology.orchestra.run.smithi046.stdout: CC engines/exec.o 2024-01-19T15:53:33.358 INFO:teuthology.orchestra.run.smithi046.stdout: CC server.o 2024-01-19T15:53:34.117 INFO:teuthology.orchestra.run.smithi046.stdout: CC client.o 2024-01-19T15:53:34.679 INFO:teuthology.orchestra.run.smithi046.stdout: CC iolog.o 2024-01-19T15:53:35.128 INFO:teuthology.orchestra.run.smithi046.stdout: CC backend.o 2024-01-19T15:53:35.991 INFO:teuthology.orchestra.run.smithi046.stdout: CC libfio.o 2024-01-19T15:53:36.182 INFO:teuthology.orchestra.run.smithi046.stdout: CC flow.o 2024-01-19T15:53:36.295 INFO:teuthology.orchestra.run.smithi046.stdout: CC cconv.o 2024-01-19T15:53:36.652 INFO:teuthology.orchestra.run.smithi046.stdout: CC gettime-thread.o 2024-01-19T15:53:36.755 INFO:teuthology.orchestra.run.smithi046.stdout: CC helpers.o 2024-01-19T15:53:36.813 INFO:teuthology.orchestra.run.smithi046.stdout: CC json.o 2024-01-19T15:53:37.001 INFO:teuthology.orchestra.run.smithi046.stdout: CC idletime.o 2024-01-19T15:53:37.232 INFO:teuthology.orchestra.run.smithi046.stdout: CC td_error.o 2024-01-19T15:53:37.330 INFO:teuthology.orchestra.run.smithi046.stdout: CC profiles/tiobench.o 2024-01-19T15:53:37.425 INFO:teuthology.orchestra.run.smithi046.stdout: CC profiles/act.o 2024-01-19T15:53:37.585 INFO:teuthology.orchestra.run.smithi046.stdout: CC io_u_queue.o 2024-01-19T15:53:37.633 INFO:teuthology.orchestra.run.smithi046.stdout: CC filelock.o 2024-01-19T15:53:37.759 INFO:teuthology.orchestra.run.smithi046.stdout: CC workqueue.o 2024-01-19T15:53:37.933 INFO:teuthology.orchestra.run.smithi046.stdout: CC rate-submit.o 2024-01-19T15:53:38.086 INFO:teuthology.orchestra.run.smithi046.stdout: CC optgroup.o 2024-01-19T15:53:38.131 INFO:teuthology.orchestra.run.smithi046.stdout: CC helper_thread.o 2024-01-19T15:53:38.311 INFO:teuthology.orchestra.run.smithi046.stdout: CC steadystate.o 2024-01-19T15:53:38.555 INFO:teuthology.orchestra.run.smithi046.stdout: CC zone-dist.o 2024-01-19T15:53:38.690 INFO:teuthology.orchestra.run.smithi046.stdout: CC zbd.o 2024-01-19T15:53:39.406 INFO:teuthology.orchestra.run.smithi046.stdout: CC dedupe.o 2024-01-19T15:53:39.552 INFO:teuthology.orchestra.run.smithi046.stdout: CC fdp.o 2024-01-19T15:53:39.696 INFO:teuthology.orchestra.run.smithi046.stdout: CC engines/posixaio.o 2024-01-19T15:53:39.828 INFO:teuthology.orchestra.run.smithi046.stdout: CC engines/falloc.o 2024-01-19T15:53:39.943 INFO:teuthology.orchestra.run.smithi046.stdout: CC engines/e4defrag.o 2024-01-19T15:53:40.073 INFO:teuthology.orchestra.run.smithi046.stdout: CC engines/splice.o 2024-01-19T15:53:40.265 INFO:teuthology.orchestra.run.smithi046.stdout: CC oslib/asprintf.o 2024-01-19T15:53:40.299 INFO:teuthology.orchestra.run.smithi046.stdout: CC oslib/strlcat.o 2024-01-19T15:53:40.341 INFO:teuthology.orchestra.run.smithi046.stdout: CC engines/mtd.o 2024-01-19T15:53:40.502 INFO:teuthology.orchestra.run.smithi046.stdout: CC oslib/libmtd.o 2024-01-19T15:53:40.887 INFO:teuthology.orchestra.run.smithi046.stdout: CC oslib/libmtd_legacy.o 2024-01-19T15:53:41.021 INFO:teuthology.orchestra.run.smithi046.stdout: CC diskutil.o 2024-01-19T15:53:41.232 INFO:teuthology.orchestra.run.smithi046.stdout: CC fifo.o 2024-01-19T15:53:41.287 INFO:teuthology.orchestra.run.smithi046.stdout: CC blktrace.o 2024-01-19T15:53:41.627 INFO:teuthology.orchestra.run.smithi046.stdout: CC cgroup.o 2024-01-19T15:53:41.777 INFO:teuthology.orchestra.run.smithi046.stdout: CC trim.o 2024-01-19T15:53:41.889 INFO:teuthology.orchestra.run.smithi046.stdout: CC engines/sg.o 2024-01-19T15:53:42.277 INFO:teuthology.orchestra.run.smithi046.stdout: CC oslib/linux-dev-lookup.o 2024-01-19T15:53:42.334 INFO:teuthology.orchestra.run.smithi046.stdout: CC engines/io_uring.o 2024-01-19T15:53:42.732 INFO:teuthology.orchestra.run.smithi046.stdout: CC engines/nvme.o 2024-01-19T15:53:43.028 INFO:teuthology.orchestra.run.smithi046.stdout: CC oslib/linux-blkzoned.o 2024-01-19T15:53:43.180 INFO:teuthology.orchestra.run.smithi046.stdout: CC engines/cmdprio.o 2024-01-19T15:53:43.422 INFO:teuthology.orchestra.run.smithi046.stdout: CC engines/rados.o 2024-01-19T15:53:43.598 INFO:teuthology.orchestra.run.smithi046.stdout: CC engines/rbd.o 2024-01-19T15:53:43.781 INFO:teuthology.orchestra.run.smithi046.stdout: CC fio.o 2024-01-19T15:53:43.868 INFO:teuthology.orchestra.run.smithi046.stdout: LINK fio 2024-01-19T15:53:44.081 INFO:teuthology.orchestra.run.smithi046.stdout: CC t/log.o 2024-01-19T15:53:44.122 INFO:teuthology.orchestra.run.smithi046.stdout: CC t/genzipf.o 2024-01-19T15:53:44.261 INFO:teuthology.orchestra.run.smithi046.stdout: CC oslib/strcasestr.o 2024-01-19T15:53:44.284 INFO:teuthology.orchestra.run.smithi046.stdout: CC oslib/strndup.o 2024-01-19T15:53:44.307 INFO:teuthology.orchestra.run.smithi046.stdout: LINK t/fio-genzipf 2024-01-19T15:53:44.398 INFO:teuthology.orchestra.run.smithi046.stdout: CC t/btrace2fio.o 2024-01-19T15:53:44.829 INFO:teuthology.orchestra.run.smithi046.stdout: LINK t/fio-btrace2fio 2024-01-19T15:53:44.920 INFO:teuthology.orchestra.run.smithi046.stdout: CC t/verify-state.o 2024-01-19T15:53:45.017 INFO:teuthology.orchestra.run.smithi046.stdout: CC t/debug.o 2024-01-19T15:53:45.050 INFO:teuthology.orchestra.run.smithi046.stdout: LINK t/fio-verify-state 2024-01-19T15:53:45.138 INFO:teuthology.orchestra.run.smithi046.stdout: CC t/stest.o 2024-01-19T15:53:45.208 INFO:teuthology.orchestra.run.smithi046.stdout: CC t/arch.o 2024-01-19T15:53:45.237 INFO:teuthology.orchestra.run.smithi046.stdout: LINK t/stest 2024-01-19T15:53:45.329 INFO:teuthology.orchestra.run.smithi046.stdout: CC t/ieee754.o 2024-01-19T15:53:45.368 INFO:teuthology.orchestra.run.smithi046.stdout: LINK t/ieee754 2024-01-19T15:53:45.456 INFO:teuthology.orchestra.run.smithi046.stdout: CC t/axmap.o 2024-01-19T15:53:45.555 INFO:teuthology.orchestra.run.smithi046.stdout: LINK t/axmap 2024-01-19T15:53:45.644 INFO:teuthology.orchestra.run.smithi046.stdout: CC t/lfsr-test.o 2024-01-19T15:53:45.739 INFO:teuthology.orchestra.run.smithi046.stdout: LINK t/lfsr-test 2024-01-19T15:53:45.830 INFO:teuthology.orchestra.run.smithi046.stdout: CC t/gen-rand.o 2024-01-19T15:53:45.926 INFO:teuthology.orchestra.run.smithi046.stdout: LINK t/gen-rand 2024-01-19T15:53:46.014 INFO:teuthology.orchestra.run.smithi046.stdout: CC t/memlock.o 2024-01-19T15:53:46.074 INFO:teuthology.orchestra.run.smithi046.stdout: LINK t/memlock 2024-01-19T15:53:46.165 INFO:teuthology.orchestra.run.smithi046.stdout: CC t/read-to-pipe-async.o 2024-01-19T15:53:46.355 INFO:teuthology.orchestra.run.smithi046.stdout: LINK t/read-to-pipe-async 2024-01-19T15:53:46.444 INFO:teuthology.orchestra.run.smithi046.stdout: CC t/io_uring.o 2024-01-19T15:53:46.995 INFO:teuthology.orchestra.run.smithi046.stdout: LINK t/io_uring 2024-01-19T15:53:47.086 DEBUG:teuthology.orchestra.run.smithi046:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-01-19T15:53:47.115 DEBUG:teuthology.orchestra.run.smithi046:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-01-19T15:53:47.343 INFO:teuthology.orchestra.run.smithi046.stderr:15:53:47 - DEBUG - cbt - Settings.cluster: 2024-01-19T15:53:47.343 INFO:teuthology.orchestra.run.smithi046.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-01-19T15:53:47.343 INFO:teuthology.orchestra.run.smithi046.stderr: 'clients': ['ubuntu@smithi046.front.sepia.ceph.com'], 2024-01-19T15:53:47.343 INFO:teuthology.orchestra.run.smithi046.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-01-19T15:53:47.344 INFO:teuthology.orchestra.run.smithi046.stderr: 'head': 'ubuntu@smithi046.front.sepia.ceph.com', 2024-01-19T15:53:47.344 INFO:teuthology.orchestra.run.smithi046.stderr: 'is_teuthology': True, 2024-01-19T15:53:47.344 INFO:teuthology.orchestra.run.smithi046.stderr: 'iterations': 1, 2024-01-19T15:53:47.344 INFO:teuthology.orchestra.run.smithi046.stderr: 'mons': ['ubuntu@smithi046.front.sepia.ceph.com'], 2024-01-19T15:53:47.344 INFO:teuthology.orchestra.run.smithi046.stderr: 'osds': ['ubuntu@smithi046.front.sepia.ceph.com'], 2024-01-19T15:53:47.345 INFO:teuthology.orchestra.run.smithi046.stderr: 'osds_per_node': 3, 2024-01-19T15:53:47.345 INFO:teuthology.orchestra.run.smithi046.stderr: 'pid_dir': '/var/run/ceph', 2024-01-19T15:53:47.345 INFO:teuthology.orchestra.run.smithi046.stderr: 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 2024-01-19T15:53:47.345 INFO:teuthology.orchestra.run.smithi046.stderr: 'rebuild_every_test': False, 2024-01-19T15:53:47.345 INFO:teuthology.orchestra.run.smithi046.stderr: 'rgws': {'ubuntu@smithi046.front.sepia.ceph.com': None}, 2024-01-19T15:53:47.345 INFO:teuthology.orchestra.run.smithi046.stderr: 'tmp_dir': '/tmp/cbt', 2024-01-19T15:53:47.345 INFO:teuthology.orchestra.run.smithi046.stderr: 'use_existing': True, 2024-01-19T15:53:47.345 INFO:teuthology.orchestra.run.smithi046.stderr: 'user': 'ubuntu'} 2024-01-19T15:53:47.346 INFO:teuthology.orchestra.run.smithi046.stderr:15:53:47 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:53:47.346 INFO:teuthology.orchestra.run.smithi046.stderr:15:53:47 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:53:47.346 INFO:teuthology.orchestra.run.smithi046.stderr:15:53:47 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:53:47.346 INFO:teuthology.orchestra.run.smithi046.stderr:15:53:47 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread; fi 2024-01-19T15:53:47.347 INFO:teuthology.orchestra.run.smithi046.stderr:15:53:47 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:53:47.347 INFO:teuthology.orchestra.run.smithi046.stderr:15:53:47 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:53:47.348 INFO:teuthology.orchestra.run.smithi046.stderr:15:53:47 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread 2024-01-19T15:53:47.349 INFO:teuthology.orchestra.run.smithi046.stderr:15:53:47 - INFO - cbt - Pausing for 60s for idle monitoring. 2024-01-19T15:53:47.350 INFO:teuthology.orchestra.run.smithi046.stderr:15:53:47 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:53:47.350 INFO:teuthology.orchestra.run.smithi046.stderr:15:53:47 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:53:47.350 INFO:teuthology.orchestra.run.smithi046.stderr:15:53:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/idle_monitoring/collectl 2024-01-19T15:53:47.352 INFO:teuthology.orchestra.run.smithi046.stderr:15:53:47 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:53:47.352 INFO:teuthology.orchestra.run.smithi046.stderr:15:53:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-01-19T15:53:47.353 INFO:teuthology.orchestra.run.smithi046.stderr:15:53:47 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:53:47.353 INFO:teuthology.orchestra.run.smithi046.stderr:15:53:47 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:53:47.354 INFO:teuthology.orchestra.run.smithi046.stderr:15:53:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/idle_monitoring/perf 2024-01-19T15:53:47.356 INFO:teuthology.orchestra.run.smithi046.stderr:15:53:47 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:53:47.356 INFO:teuthology.orchestra.run.smithi046.stderr:15:53:47 - DEBUG - cbt - PerfMonitoring: in local_node 2024-01-19T15:53:47.356 INFO:teuthology.orchestra.run.smithi046.stderr:15:53:47 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-01-19T15:54:47.370 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:54:47.371 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:54:47.371 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-01-19T15:54:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:54:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:54:47.401 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-01-19T15:54:47.437 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:54:47.438 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:54:47.438 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/* 2024-01-19T15:54:47.455 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:54:47.456 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c/$(basename ${i}).smithi046.front.sepia.ceph.com ; done 2024-01-19T15:54:47.460 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:54:47.460 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:54:47.460 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/pool_monitoring/collectl 2024-01-19T15:54:47.463 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:54:47.463 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-01-19T15:54:47.464 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:54:47.464 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:54:47.464 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/pool_monitoring/perf 2024-01-19T15:54:47.468 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:54:47.468 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - PerfMonitoring: in local_node 2024-01-19T15:54:47.468 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-01-19T15:54:47.468 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:54:47.468 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:54:47.468 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete cbt-librbdfio cbt-librbdfio --yes-i-really-really-mean-it 2024-01-19T15:54:47.774 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:54:47.774 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:54:47.774 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:47 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create cbt-librbdfio 128 128 2024-01-19T15:54:48.782 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:48 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:54:48.782 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:48 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:54:48.782 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:48 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable cbt-librbdfio rbd 2024-01-19T15:54:49.780 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:49 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:54:49.780 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:49 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:54:49.780 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:49 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio size 3 2024-01-19T15:54:50.780 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:50 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:54:50.780 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:50 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:54:50.781 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:50 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio min_size 2 2024-01-19T15:54:51.783 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:51 - INFO - cbt - Checking Health after pool creation. 2024-01-19T15:54:56.789 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:56 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:54:56.789 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:56 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:54:56.789 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:56 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-01-19T15:54:57.119 INFO:teuthology.orchestra.run.smithi046.stderr:15:54:57 - INFO - cbt - Final Pool Health Check. 2024-01-19T15:55:02.125 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:02 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:55:02.125 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:02 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:55:02.125 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-01-19T15:55:02.442 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:02 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:55:02.442 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:02 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:55:02.442 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:02 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=hostname -f 2024-01-19T15:55:02.445 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:02 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:55:02.445 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:02 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:55:02.445 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:02 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi046.front.sepia.ceph.com-0 --size 4096 --pool cbt-librbdfio --order 22 2024-01-19T15:55:04.037 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:04 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:55:04.037 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:04 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:55:04.037 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:04 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi046.front.sepia.ceph.com-1 --size 4096 --pool cbt-librbdfio --order 22 2024-01-19T15:55:04.243 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:04 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:55:04.243 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:04 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:55:04.243 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-01-19T15:55:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:04 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:55:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:04 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:55:04.260 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-01-19T15:55:04.289 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:04 - INFO - cbt - Attempting to populating fio files... 2024-01-19T15:55:04.289 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:04 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:55:04.289 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:04 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:55:04.289 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2024-01-19T15:55:04.291 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:04 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:55:04.291 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:04 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:55:04.291 INFO:teuthology.orchestra.run.smithi046.stderr:15:55:04 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2024-01-19T15:57:38.027 INFO:teuthology.orchestra.run.smithi046.stderr:15:57:38 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:57:38.027 INFO:teuthology.orchestra.run.smithi046.stderr:15:57:38 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c. 2024-01-19T15:57:38.027 INFO:teuthology.orchestra.run.smithi046.stderr:15:57:38 - INFO - cbt - Setting OSD Read Ahead to: 4096 2024-01-19T15:57:38.027 INFO:teuthology.orchestra.run.smithi046.stderr:15:57:38 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:57:38.027 INFO:teuthology.orchestra.run.smithi046.stderr:15:57:38 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:57:38.028 INFO:teuthology.orchestra.run.smithi046.stderr:15:57:38 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 4096 > /sys/block/'{}'/queue/read_ahead_kb" 2024-01-19T15:57:38.028 INFO:teuthology.orchestra.run.smithi046.stderr:15:57:38 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread 2024-01-19T15:57:38.028 INFO:teuthology.orchestra.run.smithi046.stderr:15:57:38 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:57:38.028 INFO:teuthology.orchestra.run.smithi046.stderr:15:57:38 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:57:38.029 INFO:teuthology.orchestra.run.smithi046.stderr:15:57:38 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread 2024-01-19T15:57:38.043 INFO:teuthology.orchestra.run.smithi046.stderr:15:57:38 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:57:38.043 INFO:teuthology.orchestra.run.smithi046.stderr:15:57:38 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:57:38.044 INFO:teuthology.orchestra.run.smithi046.stderr:15:57:38 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-01-19T15:57:38.772 INFO:teuthology.orchestra.run.smithi046.stderr:15:57:38 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:57:38.773 INFO:teuthology.orchestra.run.smithi046.stderr:15:57:38 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-01-19T15:57:39.508 INFO:teuthology.orchestra.run.smithi046.stderr:15:57:39 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:57:39.509 INFO:teuthology.orchestra.run.smithi046.stderr:15:57:39 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:57:39.509 INFO:teuthology.orchestra.run.smithi046.stderr:15:57:39 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread 2024-01-19T15:57:39.620 INFO:teuthology.orchestra.run.smithi046.stderr:15:57:39 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:57:39.620 INFO:teuthology.orchestra.run.smithi046.stderr:15:57:39 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:57:39.620 INFO:teuthology.orchestra.run.smithi046.stderr:15:57:39 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/ceph_settings.out 2024-01-19T15:58:05.521 INFO:teuthology.orchestra.run.smithi046.stderr:15:58:05 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:58:05.521 INFO:teuthology.orchestra.run.smithi046.stderr:15:58:05 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:58:05.521 INFO:teuthology.orchestra.run.smithi046.stderr:15:58:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo "[`date`] ceph progress" >> /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/pgautoscaler.log;/usr/bin/ceph -c /etc/ceph/ceph.conf progress | tee -a /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/pgautoscaler.log 2024-01-19T15:58:05.887 INFO:teuthology.orchestra.run.smithi046.stderr:15:58:05 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:58:05.887 INFO:teuthology.orchestra.run.smithi046.stderr:15:58:05 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:58:05.887 INFO:teuthology.orchestra.run.smithi046.stderr:15:58:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/collectl 2024-01-19T15:58:05.890 INFO:teuthology.orchestra.run.smithi046.stderr:15:58:05 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:58:05.890 INFO:teuthology.orchestra.run.smithi046.stderr:15:58:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-01-19T15:58:05.892 INFO:teuthology.orchestra.run.smithi046.stderr:15:58:05 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:58:05.892 INFO:teuthology.orchestra.run.smithi046.stderr:15:58:05 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:58:05.892 INFO:teuthology.orchestra.run.smithi046.stderr:15:58:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/perf 2024-01-19T15:58:05.896 INFO:teuthology.orchestra.run.smithi046.stderr:15:58:05 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:58:05.896 INFO:teuthology.orchestra.run.smithi046.stderr:15:58:05 - DEBUG - cbt - PerfMonitoring: in local_node 2024-01-19T15:58:05.896 INFO:teuthology.orchestra.run.smithi046.stderr:15:58:05 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-01-19T15:58:05.896 INFO:teuthology.orchestra.run.smithi046.stderr:15:58:05 - INFO - cbt - Running rbd fio randread test. 2024-01-19T15:58:05.896 INFO:teuthology.orchestra.run.smithi046.stderr:15:58:05 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-0 2024-01-19T15:58:05.896 INFO:teuthology.orchestra.run.smithi046.stderr:15:58:05 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:58:05.896 INFO:teuthology.orchestra.run.smithi046.stderr:15:58:05 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:58:05.896 INFO:teuthology.orchestra.run.smithi046.stderr:15:58:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=randread --output-format=json,normal --runtime=60 --numjobs=1 --direct=1 --bs=4096B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.0 2024-01-19T15:58:05.897 INFO:teuthology.orchestra.run.smithi046.stderr:15:58:05 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-1 2024-01-19T15:58:05.898 INFO:teuthology.orchestra.run.smithi046.stderr:15:58:05 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:58:05.898 INFO:teuthology.orchestra.run.smithi046.stderr:15:58:05 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:58:05.898 INFO:teuthology.orchestra.run.smithi046.stderr:15:58:05 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=randread --output-format=json,normal --runtime=60 --numjobs=1 --direct=1 --bs=4096B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.1 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/output.1 2024-01-19T15:59:06.822 INFO:teuthology.orchestra.run.smithi046.stderr:15:59:06 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:59:06.822 INFO:teuthology.orchestra.run.smithi046.stderr:15:59:06 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:59:06.822 INFO:teuthology.orchestra.run.smithi046.stderr:15:59:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-01-19T15:59:06.842 INFO:teuthology.orchestra.run.smithi046.stderr:15:59:06 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:59:06.843 INFO:teuthology.orchestra.run.smithi046.stderr:15:59:06 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:59:06.843 INFO:teuthology.orchestra.run.smithi046.stderr:15:59:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-01-19T15:59:06.872 INFO:teuthology.orchestra.run.smithi046.stderr:15:59:06 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:59:06.872 INFO:teuthology.orchestra.run.smithi046.stderr:15:59:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/perf/perf.data 2024-01-19T15:59:06.874 INFO:teuthology.orchestra.run.smithi046.stderr:15:59:06 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:59:06.874 INFO:teuthology.orchestra.run.smithi046.stderr:15:59:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/perf/perf_stat.* 2024-01-19T15:59:06.875 INFO:teuthology.orchestra.run.smithi046.stderr:15:59:06 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:59:06.875 INFO:teuthology.orchestra.run.smithi046.stderr:15:59:06 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:59:06.875 INFO:teuthology.orchestra.run.smithi046.stderr:15:59:06 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/historic_ops.out 2024-01-19T15:59:07.171 INFO:teuthology.orchestra.run.smithi046.stderr:15:59:07 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:59:07.191 INFO:teuthology.orchestra.run.smithi046.stderr:15:59:07 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:59:07.191 INFO:teuthology.orchestra.run.smithi046.stderr:15:59:07 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/* 2024-01-19T15:59:07.207 INFO:teuthology.orchestra.run.smithi046.stderr:15:59:07 - DEBUG - cbt - get_fqdn_local()=smithi046.front.sepia.ceph.com 2024-01-19T15:59:07.207 INFO:teuthology.orchestra.run.smithi046.stderr:15:59:07 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c/$(basename ${i}).smithi046.front.sepia.ceph.com ; done 2024-01-19T15:59:07.234 INFO:teuthology.orchestra.run.smithi046.stderr:15:59:07 - INFO - cbt - Convert results to json format. 2024-01-19T15:59:07.234 INFO:teuthology.orchestra.run.smithi046.stderr:15:59:07 - DEBUG - cbt - Nodes : ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:59:07.234 INFO:teuthology.orchestra.run.smithi046.stdout:cleaning remote dir /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randread 2024-01-19T15:59:07.234 INFO:teuthology.orchestra.run.smithi046.stdout:smithi046.front.sepia.ceph.com: smithi046.front.sepia.ceph.com 2024-01-19T15:59:07.234 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-01-19T15:59:07.234 INFO:teuthology.orchestra.run.smithi046.stdout:['smithi046.front.sepia.ceph.com'] 2024-01-19T15:59:07.256 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2024-01-19T15:59:07.268 DEBUG:teuthology.orchestra.run.smithi046:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2024-01-19T15:59:07.436 DEBUG:teuthology.orchestra.run.smithi046:> rm --one-file-system -rf -- /home/ubuntu/cephtest/fio 2024-01-19T15:59:07.485 INFO:tasks.cbt_performance:Collecting CBT performance data config 2024-01-19T15:59:07.485 INFO:tasks.cbt_performance:reading cbt preferences 2024-01-19T15:59:07.485 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest -name 'json_output.*' 2024-01-19T15:59:07.547 INFO:teuthology.orchestra.run.smithi046.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c/json_output.1.smithi046.front.sepia.ceph.com 2024-01-19T15:59:07.547 INFO:teuthology.orchestra.run.smithi046.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c/json_output.0.smithi046.front.sepia.ceph.com 2024-01-19T15:59:07.548 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-19T15:59:07.548 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c/json_output.1.smithi046.front.sepia.ceph.com of=/dev/stdout 2024-01-19T15:59:07.564 INFO:tasks.cbt_performance:reading total cpu cycles 2024-01-19T15:59:07.564 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c -name 'perf_stat.*' 2024-01-19T15:59:07.618 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-01-19T15:59:07.618 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-01-19T15:59:07.618 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-01-19T15:59:07.619 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c/json_output.0.smithi046.front.sepia.ceph.com of=/dev/stdout 2024-01-19T15:59:07.674 INFO:tasks.cbt_performance:reading total cpu cycles 2024-01-19T15:59:07.674 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-c9606d7c -name 'perf_stat.*' 2024-01-19T15:59:07.728 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-01-19T15:59:07.729 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-01-19T15:59:07.739 INFO:tasks.cbt_performance:Data inserted successfully. 2024-01-19T15:59:07.747 INFO:tasks.cbt_performance:Data inserted successfully. 2024-01-19T15:59:07.748 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-19T15:59:07.758 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-01-19T15:59:07.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:59:07.879+0000 7fa77b0f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:59:07.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-01-19T15:59:07.886+0000 7fa77b0f0640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:59:08.079 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-19T15:59:08.079 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-19T15:59:08.079 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-19T15:59:08.080 INFO:teuthology.orchestra.run:waiting for 300 2024-01-19T15:59:11.455 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-01-19T15:59:14.181 INFO:tasks.ceph.osd.0:Stopped 2024-01-19T15:59:14.181 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-19T15:59:14.181 INFO:teuthology.orchestra.run:waiting for 300 2024-01-19T15:59:16.758 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-01-19T15:59:16.758 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-01-19T15:59:20.283 INFO:tasks.ceph.osd.1:Stopped 2024-01-19T15:59:20.284 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-19T15:59:20.284 INFO:teuthology.orchestra.run:waiting for 300 2024-01-19T15:59:21.960 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-01-19T15:59:21.961 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-01-19T15:59:26.386 INFO:tasks.ceph.osd.2:Stopped 2024-01-19T15:59:26.386 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-19T15:59:26.386 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-19T15:59:26.386 INFO:teuthology.orchestra.run:waiting for 300 2024-01-19T15:59:26.419 INFO:tasks.ceph.mgr.x:Stopped 2024-01-19T15:59:26.419 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-19T15:59:26.420 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-19T15:59:26.420 INFO:teuthology.orchestra.run:waiting for 300 2024-01-19T15:59:26.472 INFO:tasks.ceph.mon.a:Stopped 2024-01-19T15:59:26.472 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-19T15:59:26.472 DEBUG:teuthology.orchestra.run.smithi046:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v 'overall HEALTH' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-01-19T15:59:26.567 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:59:26.567 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-01-19T15:59:26.808 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:59:26.808 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-01-19T15:59:26.923 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:59:26.923 DEBUG:teuthology.orchestra.run.smithi046:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-01-19T15:59:26.961 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-01-19T15:59:27.125 INFO:tasks.ceph:Archiving mon data... 2024-01-19T15:59:27.126 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yingxin-2024-01-19_07:32:18-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-8-distro-default-smithi/7521657/data/mon.a.tgz 2024-01-19T15:59:27.126 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2024-01-19T15:59:27.148 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.q0SgvhmEnx 2024-01-19T15:59:27.148 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.q0SgvhmEnx 2024-01-19T15:59:27.277 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.q0SgvhmEnx 2024-01-19T15:59:27.376 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.q0SgvhmEnx is 205KB 2024-01-19T15:59:27.442 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.q0SgvhmEnx 2024-01-19T15:59:27.456 INFO:tasks.ceph:Cleaning ceph cluster... 2024-01-19T15:59:27.456 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-19T15:59:27.577 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-01-19T15:59:27.577 INFO:tasks.ceph:Archiving crash dumps... 2024-01-19T15:59:27.579 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2024-01-19_07:32:18-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-8-distro-default-smithi/7521657/remote/smithi046/crash 2024-01-19T15:59:27.580 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-19T15:59:27.643 INFO:tasks.ceph:Compressing logs... 2024-01-19T15:59:27.643 DEBUG:teuthology.orchestra.run.smithi046:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-19T15:59:27.725 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29692.log 2024-01-19T15:59:27.725 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-01-19T15:59:27.725 INFO:teuthology.orchestra.run.smithi046.stderr:gzip/var/log/ceph/ceph-client.admin.29692.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34284.log 2024-01-19T15:59:27.725 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30452.log 2024-01-19T15:59:27.725 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.34284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30834.log 2024-01-19T15:59:27.725 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.30452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34283.log 2024-01-19T15:59:27.726 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.30834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34235.log 2024-01-19T15:59:27.726 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.34283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34252.log 2024-01-19T15:59:27.726 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.34235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30185.log 2024-01-19T15:59:27.726 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.34252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34165.log 2024-01-19T15:59:27.726 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.30185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30129.log 2024-01-19T15:59:27.726 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.34165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29722.log 2024-01-19T15:59:27.726 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.30129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34024.log 2024-01-19T15:59:27.726 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29722.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27897.log 2024-01-19T15:59:27.726 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.34024.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-19T15:59:27.726 INFO:teuthology.orchestra.run.smithi046.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29692.log.gz 2024-01-19T15:59:27.726 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.tmp-client.admin.27897.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34284.log.gz 2024-01-19T15:59:27.726 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30288.log 2024-01-19T15:59:27.727 INFO:teuthology.orchestra.run.smithi046.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34283.log.gz 2024-01-19T15:59:27.727 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30792.log 2024-01-19T15:59:27.727 INFO:teuthology.orchestra.run.smithi046.stderr: 66.4% -- replaced with /var/log/ceph/ceph-client.admin.34235.log.gz 2024-01-19T15:59:27.727 INFO:teuthology.orchestra.run.smithi046.stderr: 65.9% -- replaced with /var/log/ceph/ceph-client.admin.34252.log.gz 2024-01-19T15:59:27.727 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.30288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34115.log 2024-01-19T15:59:27.727 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.30792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30330.log 2024-01-19T15:59:27.727 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30372.log 2024-01-19T15:59:27.727 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.34115.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29722.log.gz 2024-01-19T15:59:27.727 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.30330.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-01-19T15:59:27.727 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.30372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34140.log 2024-01-19T15:59:27.727 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29494.log 2024-01-19T15:59:27.727 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.34140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34883.log 2024-01-19T15:59:27.727 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29344.log 2024-01-19T15:59:27.727 INFO:teuthology.orchestra.run.smithi046.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27897.log.gz 2024-01-19T15:59:27.727 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29256.log 2024-01-19T15:59:27.728 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.34883.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34883.log.gz 2024-01-19T15:59:27.728 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30750.log 2024-01-19T15:59:27.728 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30454.log 2024-01-19T15:59:27.728 INFO:teuthology.orchestra.run.smithi046.stderr: 85.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-19T15:59:27.728 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-01-19T15:59:27.728 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.30750.log: /var/log/ceph/ceph-client.admin.30454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30227.log 2024-01-19T15:59:27.728 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-01-19T15:59:27.728 INFO:teuthology.orchestra.run.smithi046.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29344.log.gz 2024-01-19T15:59:27.728 INFO:teuthology.orchestra.run.smithi046.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29256.log.gz 2024-01-19T15:59:27.728 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30666.log 2024-01-19T15:59:27.728 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.30227.log: /var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30540.log 2024-01-19T15:59:27.728 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.30666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34547.log 2024-01-19T15:59:27.728 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.30540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30580.log 2024-01-19T15:59:27.728 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-19T15:59:27.729 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.34547.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34547.log.gz 2024-01-19T15:59:27.729 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.30580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30453.log 2024-01-19T15:59:27.729 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-01-19T15:59:27.729 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.30453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34211.log 2024-01-19T15:59:27.729 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34594.log 2024-01-19T15:59:27.729 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.34211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34088.log 2024-01-19T15:59:27.729 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.34594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34595.log 2024-01-19T15:59:27.729 INFO:teuthology.orchestra.run.smithi046.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.34594.log.gz 2024-01-19T15:59:27.729 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.34088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34188.log 2024-01-19T15:59:27.729 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.34595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29454.log 2024-01-19T15:59:27.729 INFO:teuthology.orchestra.run.smithi046.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34595.log.gz 2024-01-19T15:59:27.729 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29534.log 2024-01-19T15:59:27.729 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.34188.log: /var/log/ceph/ceph-client.admin.29454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30579.log 2024-01-19T15:59:27.729 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.29534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30708.log 2024-01-19T15:59:27.730 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.30579.log: /var/log/ceph/ceph-client.admin.30708.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30792.log.gz 2024-01-19T15:59:27.730 INFO:teuthology.orchestra.run.smithi046.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30750.log.gz 2024-01-19T15:59:27.731 INFO:teuthology.orchestra.run.smithi046.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34211.log.gz 2024-01-19T15:59:27.731 INFO:teuthology.orchestra.run.smithi046.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34188.log.gz 2024-01-19T15:59:27.731 INFO:teuthology.orchestra.run.smithi046.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30708.log.gz 2024-01-19T15:59:27.731 INFO:teuthology.orchestra.run.smithi046.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30834.log.gz 2024-01-19T15:59:27.734 INFO:teuthology.orchestra.run.smithi046.stderr: 87.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-19T15:59:27.762 INFO:teuthology.orchestra.run.smithi046.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29454.log.gz 2024-01-19T15:59:27.763 INFO:teuthology.orchestra.run.smithi046.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29494.log.gz 2024-01-19T15:59:27.763 INFO:teuthology.orchestra.run.smithi046.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29534.log.gz 2024-01-19T15:59:27.769 INFO:teuthology.orchestra.run.smithi046.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30129.log.gz 2024-01-19T15:59:27.769 INFO:teuthology.orchestra.run.smithi046.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30185.log.gz 2024-01-19T15:59:27.769 INFO:teuthology.orchestra.run.smithi046.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30227.log.gz 2024-01-19T15:59:27.769 INFO:teuthology.orchestra.run.smithi046.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30288.log.gz 2024-01-19T15:59:27.769 INFO:teuthology.orchestra.run.smithi046.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30330.log.gz 2024-01-19T15:59:27.770 INFO:teuthology.orchestra.run.smithi046.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30372.log.gz 2024-01-19T15:59:27.770 INFO:teuthology.orchestra.run.smithi046.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30452.log.gz 2024-01-19T15:59:27.770 INFO:teuthology.orchestra.run.smithi046.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.30453.log.gz 2024-01-19T15:59:27.770 INFO:teuthology.orchestra.run.smithi046.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30454.log.gz 2024-01-19T15:59:27.770 INFO:teuthology.orchestra.run.smithi046.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30540.log.gz 2024-01-19T15:59:27.770 INFO:teuthology.orchestra.run.smithi046.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30580.log.gz 2024-01-19T15:59:27.770 INFO:teuthology.orchestra.run.smithi046.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30579.log.gz 2024-01-19T15:59:27.770 INFO:teuthology.orchestra.run.smithi046.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30666.log.gz 2024-01-19T15:59:27.774 INFO:teuthology.orchestra.run.smithi046.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34024.log.gz 2024-01-19T15:59:27.774 INFO:teuthology.orchestra.run.smithi046.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34088.log.gz 2024-01-19T15:59:27.774 INFO:teuthology.orchestra.run.smithi046.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34115.log.gz 2024-01-19T15:59:27.774 INFO:teuthology.orchestra.run.smithi046.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34140.log.gz 2024-01-19T15:59:27.774 INFO:teuthology.orchestra.run.smithi046.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34165.log.gz 2024-01-19T15:59:27.774 INFO:teuthology.orchestra.run.smithi046.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-01-19T15:59:27.837 INFO:teuthology.orchestra.run.smithi046.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-01-19T15:59:44.481 INFO:teuthology.orchestra.run.smithi046.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-01-19T15:59:44.640 INFO:teuthology.orchestra.run.smithi046.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-01-19T15:59:44.755 INFO:teuthology.orchestra.run.smithi046.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-01-19T15:59:44.756 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-01-19T15:59:44.756 INFO:teuthology.orchestra.run.smithi046.stderr:real 0m17.059s 2024-01-19T15:59:44.757 INFO:teuthology.orchestra.run.smithi046.stderr:user 0m9.931s 2024-01-19T15:59:44.757 INFO:teuthology.orchestra.run.smithi046.stderr:sys 0m0.534s 2024-01-19T15:59:44.757 INFO:tasks.ceph:Archiving logs... 2024-01-19T15:59:44.757 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/log/ceph to /home/teuthworker/archive/yingxin-2024-01-19_07:32:18-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-8-distro-default-smithi/7521657/remote/smithi046/log 2024-01-19T15:59:44.758 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-19T15:59:46.281 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-19T15:59:46.290 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-19T15:59:46.290 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-19T15:59:46.347 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-19T15:59:46.357 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-19T15:59:46.357 DEBUG:teuthology.orchestra.run.smithi046:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-19T15:59:46.375 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: ntpq: command not found 2024-01-19T15:59:46.435 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-19T15:59:46.435 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-01-19T15:59:46.435 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 17 -406us[ -406us] +/- 74ms 2024-01-19T15:59:46.435 INFO:teuthology.orchestra.run.smithi046.stdout:^* hv02.front.sepia.ceph.com 3 6 377 18 -1506us[-1382us] +/- 51ms 2024-01-19T15:59:46.436 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 +2479us[+2603us] +/- 70ms 2024-01-19T15:59:46.436 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T15:59:46.462 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-19T15:59:46.471 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-19T15:59:46.472 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-19T15:59:46.482 DEBUG:teuthology.orchestra.run.smithi046:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-19T15:59:46.644 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-19T15:59:46.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-19T15:59:46.673 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-19T15:59:46.682 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-19T15:59:46.692 INFO:teuthology.task.internal:Duration was 926.750644 seconds 2024-01-19T15:59:46.693 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-19T15:59:46.703 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-19T15:59:46.704 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-19T15:59:46.822 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-19T15:59:47.413 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-19T15:59:47.414 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi046.front.sepia.ceph.com 2024-01-19T15:59:47.414 DEBUG:teuthology.orchestra.run.smithi046:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-19T15:59:47.433 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-19T15:59:47.433 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-19T15:59:47.504 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-19T15:59:47.504 DEBUG:teuthology.orchestra.run.smithi046:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-19T15:59:47.665 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-19T15:59:47.675 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-19T15:59:47.675 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-19T15:59:47.713 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-19T15:59:47.724 DEBUG:teuthology.orchestra.run.smithi046:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-19T15:59:47.796 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = core 2024-01-19T15:59:47.812 DEBUG:teuthology.orchestra.run.smithi046:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-19T15:59:47.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-19T15:59:47.867 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-19T15:59:47.878 INFO:teuthology.task.internal:Transferring archived files... 2024-01-19T15:59:47.879 DEBUG:teuthology.misc:Transferring archived files from smithi046:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yingxin-2024-01-19_07:32:18-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-8-distro-default-smithi/7521657/remote/smithi046 2024-01-19T15:59:47.879 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-19T15:59:47.976 INFO:teuthology.task.internal:Removing archive directory... 2024-01-19T15:59:47.976 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-19T15:59:48.020 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-19T15:59:48.033 INFO:teuthology.task.internal:Not uploading archives. 2024-01-19T15:59:48.033 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-19T15:59:48.043 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-19T15:59:48.043 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-19T15:59:48.074 INFO:teuthology.orchestra.run.smithi046.stdout: 265301 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 19 15:59 /home/ubuntu/cephtest 2024-01-19T15:59:48.075 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-19T15:59:48.086 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-19T15:59:48.111 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/2Jx1MmfVk/fio?orgId=1&var-branch_name=wip-yingxin-crimson-save-conn-foreign-copy-8&var-job_id_selected=7521657 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} duration: 926.7506442070007 flavor: crimson owner: scheduled_yingxin@teuthology success: true 2024-01-19T15:59:48.111 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-19T15:59:48.180 INFO:teuthology.run:pass