2024-01-19T15:43:28.544 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-19T15:43:28.545 DEBUG:teuthology.run:Teuthology command: teuthology --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read} --name yingxin-2024-01-19_07:32:18-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-8-distro-default-smithi --owner scheduled_yingxin@teuthology --verbose --archive /home/teuthworker/archive/yingxin-2024-01-19_07:32:18-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-8-distro-default-smithi/7521663 -- /home/teuthworker/archive/yingxin-2024-01-19_07:32:18-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-8-distro-default-smithi/7521663/orig.config.yaml 2024-01-19T15:43:28.590 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-19T15:43:28.678 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yingxin-2024-01-19_07:32:18-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-8-distro-default-smithi/7521663 branch: wip-yingxin-crimson-save-conn-foreign-copy-8 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read} email: yingxin.cheng@intel.com first_in_suite: false job_id: '7521663' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yingxin-2024-01-19_07:32:18-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-8-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yingxin-crimson-save-conn-foreign-copy-8 ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson osd obc lru size: 10 crimson seastar smp: 3 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4 workunit: branch: wip-yingxin-crimson-save-conn-foreign-copy-8 env: CRIMSON_COMPAT: '1' sha1: bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4 owner: scheduled_yingxin@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4 sleep_before_teardown: 0 suite: crimson-rados suite_branch: wip-yingxin-crimson-save-conn-foreign-copy-8 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4 targets: smithi181.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNoUEJRLjb2j1PKdTbNzHU5CltHfFrjhnEZwFVvTj+KEbI/zu9HTGnalyR4eiHbcaehdOG9zc+189044MkncjLI= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: librbdfio: iodepth: - 32 log_avg_msec: 100 mode: - randread norandommap: true op_size: - 4194304 osd_ra: - 4096 pool_profile: rbd procs_per_volume: - 1 time: 60 vol_size: 4096 volumes_per_client: - 2 cluster: iterations: 1 osds_per_node: 3 pool_profiles: rbd: pg_size: 128 pgp_size: 128 replication: 3 user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-19_07:32:18 tube: smithi user: yingxin verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-19T15:43:28.679 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4/qa; will attempt to use it 2024-01-19T15:43:28.679 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4/qa/tasks 2024-01-19T15:43:28.679 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-19T15:43:28.681 INFO:teuthology.task.internal:Checking packages... 2024-01-19T15:43:28.705 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash 'bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4' 2024-01-19T15:43:28.706 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-19T15:43:28.706 INFO:teuthology.packaging:ref: None 2024-01-19T15:43:28.706 INFO:teuthology.packaging:tag: None 2024-01-19T15:43:28.706 INFO:teuthology.packaging:branch: wip-yingxin-crimson-save-conn-foreign-copy-8 2024-01-19T15:43:28.706 INFO:teuthology.packaging:sha1: bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4 2024-01-19T15:43:28.706 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=wip-yingxin-crimson-save-conn-foreign-copy-8 2024-01-19T15:43:28.928 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-767.gbdd6070c 2024-01-19T15:43:28.929 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-19T15:43:28.938 INFO:teuthology.task.internal:no buildpackages task found 2024-01-19T15:43:28.938 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-19T15:43:28.962 INFO:teuthology.task.internal:Saving configuration 2024-01-19T15:43:28.979 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-19T15:43:28.990 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-19T15:43:29.012 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-01-19_07:32:18-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-8-distro-default-smithi/7521663', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-19 15:36:33.034199', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNoUEJRLjb2j1PKdTbNzHU5CltHfFrjhnEZwFVvTj+KEbI/zu9HTGnalyR4eiHbcaehdOG9zc+189044MkncjLI='} 2024-01-19T15:43:29.012 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-19T15:43:29.022 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-01-19T15:43:29.022 INFO:teuthology.run_tasks:Running task console_log... 2024-01-19T15:43:29.063 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f90835d5dc0>, signals=[15]) 2024-01-19T15:43:29.063 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-19T15:43:29.069 INFO:teuthology.task.internal:Opening connections... 2024-01-19T15:43:29.069 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:43:29.071 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T15:43:29.202 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-19T15:43:29.208 DEBUG:teuthology.orchestra.run.smithi181:> uname -m 2024-01-19T15:43:29.269 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2024-01-19T15:43:29.269 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2024-01-19T15:43:29.326 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="CentOS Stream" 2024-01-19T15:43:29.326 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="9" 2024-01-19T15:43:29.326 INFO:teuthology.orchestra.run.smithi181.stdout:ID="centos" 2024-01-19T15:43:29.326 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="rhel fedora" 2024-01-19T15:43:29.326 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="9" 2024-01-19T15:43:29.326 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el9" 2024-01-19T15:43:29.326 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-19T15:43:29.326 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31" 2024-01-19T15:43:29.327 INFO:teuthology.orchestra.run.smithi181.stdout:LOGO="fedora-logo-icon" 2024-01-19T15:43:29.327 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-19T15:43:29.327 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://centos.org/" 2024-01-19T15:43:29.327 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-19T15:43:29.327 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-19T15:43:29.327 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-19T15:43:29.327 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2024-01-19T15:43:29.353 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-19T15:43:29.369 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-19T15:43:29.399 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-19T15:43:29.399 DEBUG:teuthology.orchestra.run.smithi181:> test '!' -e /home/ubuntu/cephtest 2024-01-19T15:43:29.415 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-19T15:43:29.426 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-19T15:43:29.426 DEBUG:teuthology.orchestra.run.smithi181:> test -z $(ls -A /var/lib/ceph) 2024-01-19T15:43:29.590 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-19T15:43:29.629 INFO:teuthology.run_tasks:Running task kernel... 2024-01-19T15:43:29.647 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-19T15:43:29.647 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-19T15:43:29.648 DEBUG:teuthology.orchestra.run.smithi181:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-19T15:43:29.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-19T15:43:29.664 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-01-19T15:43:29.722 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-361.el9.x86_64 2024-01-19T15:43:29.722 INFO:teuthology.task.kernel:Running kernel on smithi181: 5.14.0-361.el9.x86_64 2024-01-19T15:43:29.722 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum install -y kernel 2024-01-19T15:43:31.900 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS-9 - AppStream 11 MB/s | 6.4 MB 00:00 2024-01-19T15:43:33.526 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-01-19T15:43:37.699 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-01-19T15:43:40.567 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 10 MB/s | 19 MB 00:01 2024-01-19T15:43:44.617 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 9.2 MB/s | 5.9 MB 00:00 2024-01-19T15:43:46.426 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 21 kB/s | 15 kB 00:00 2024-01-19T15:43:47.439 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 23 MB/s | 20 MB 00:00 2024-01-19T15:43:52.236 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-01-19T15:43:54.426 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-01-19T15:43:54.426 INFO:teuthology.orchestra.run.smithi181.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-01-19T15:43:54.538 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:43:54.543 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:43:54.544 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-01-19T15:43:54.544 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:43:54.544 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-01-19T15:43:54.544 INFO:teuthology.orchestra.run.smithi181.stdout: kernel x86_64 5.14.0-409.el9 CentOS-BaseOS 6.0 M 2024-01-19T15:43:54.544 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-01-19T15:43:54.544 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core x86_64 5.14.0-409.el9 CentOS-BaseOS 21 M 2024-01-19T15:43:54.544 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules x86_64 5.14.0-409.el9 CentOS-BaseOS 39 M 2024-01-19T15:43:54.544 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-core x86_64 5.14.0-409.el9 CentOS-BaseOS 33 M 2024-01-19T15:43:54.544 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:43:54.544 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-01-19T15:43:54.545 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:43:54.545 INFO:teuthology.orchestra.run.smithi181.stdout:Install 4 Packages 2024-01-19T15:43:54.545 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:43:54.545 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 99 M 2024-01-19T15:43:54.545 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 122 M 2024-01-19T15:43:54.545 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-01-19T15:43:54.912 INFO:teuthology.orchestra.run.smithi181.stdout:(1/4): kernel-5.14.0-409.el9.x86_64.rpm 16 MB/s | 6.0 MB 00:00 2024-01-19T15:43:55.396 INFO:teuthology.orchestra.run.smithi181.stdout:(2/4): kernel-core-5.14.0-409.el9.x86_64.rpm 24 MB/s | 21 MB 00:00 2024-01-19T15:43:56.087 INFO:teuthology.orchestra.run.smithi181.stdout:(3/4): kernel-modules-5.14.0-409.el9.x86_64.rpm 25 MB/s | 39 MB 00:01 2024-01-19T15:43:56.504 INFO:teuthology.orchestra.run.smithi181.stdout:(4/4): kernel-modules-core-5.14.0-409.el9.x86_6 21 MB/s | 33 MB 00:01 2024-01-19T15:43:56.507 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:43:56.508 INFO:teuthology.orchestra.run.smithi181.stdout:Total 51 MB/s | 99 MB 00:01 2024-01-19T15:43:56.508 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-01-19T15:43:56.861 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-01-19T15:43:56.861 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-01-19T15:43:57.893 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-01-19T15:43:57.894 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-01-19T15:44:00.290 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-01-19T15:44:01.008 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-modules-core-5.14.0-409.el9.x86_64 1/4 2024-01-19T15:44:01.146 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-core-5.14.0-409.el9.x86_64 2/4 2024-01-19T15:44:03.905 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-5.14.0-409.el9.x86_64 2/4 2024-01-19T15:44:04.091 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-modules-5.14.0-409.el9.x86_64 3/4 2024-01-19T15:44:08.759 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-5.14.0-409.el9.x86_64 3/4 2024-01-19T15:44:08.936 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : kernel-5.14.0-409.el9.x86_64 4/4 2024-01-19T15:44:13.662 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-core-5.14.0-409.el9.x86_64 4/4 2024-01-19T15:44:42.942 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-core-5.14.0-409.el9.x86_64 4/4 2024-01-19T15:44:42.942 INFO:teuthology.orchestra.run.smithi181.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-19T15:44:42.942 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:44:43.012 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-modules-5.14.0-409.el9.x86_64 4/4 2024-01-19T15:44:43.429 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: kernel-5.14.0-409.el9.x86_64 4/4 2024-01-19T15:44:43.429 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-5.14.0-409.el9.x86_64 1/4 2024-01-19T15:44:43.429 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-core-5.14.0-409.el9.x86_64 2/4 2024-01-19T15:44:43.429 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-modules-5.14.0-409.el9.x86_64 3/4 2024-01-19T15:44:43.855 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : kernel-modules-core-5.14.0-409.el9.x86_64 4/4 2024-01-19T15:44:43.855 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:44:43.855 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-01-19T15:44:43.855 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-5.14.0-409.el9.x86_64 2024-01-19T15:44:43.855 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-core-5.14.0-409.el9.x86_64 2024-01-19T15:44:43.855 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-5.14.0-409.el9.x86_64 2024-01-19T15:44:43.856 INFO:teuthology.orchestra.run.smithi181.stdout: kernel-modules-core-5.14.0-409.el9.x86_64 2024-01-19T15:44:43.856 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:44:43.856 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:44:44.366 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-01-19T15:44:44.498 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-409.el9.x86_64 2024-01-19T15:44:44.498 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-409.el9.x86_64 2024-01-19T15:44:44.498 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-409.el9.x86_64 2024-01-19T15:44:44.498 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-19T15:44:44.498 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-19T15:44:44.498 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-01-19T15:44:44.498 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi181.front.sepia.ceph.com, path=None, version=distro) 2024-01-19T15:44:44.499 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q kernel | sort -rV | head -n 1 2024-01-19T15:44:44.607 INFO:teuthology.orchestra.run.smithi181.stdout:kernel-5.14.0-409.el9.x86_64 2024-01-19T15:44:44.607 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-409.el9.x86_64 2024-01-19T15:44:44.607 DEBUG:teuthology.orchestra.run.smithi181:> sudo rpm -qi grub2-tools 2024-01-19T15:44:44.641 INFO:teuthology.orchestra.run.smithi181.stdout:Name : grub2-tools 2024-01-19T15:44:44.641 INFO:teuthology.orchestra.run.smithi181.stdout:Epoch : 1 2024-01-19T15:44:44.642 INFO:teuthology.orchestra.run.smithi181.stdout:Version : 2.06 2024-01-19T15:44:44.642 INFO:teuthology.orchestra.run.smithi181.stdout:Release : 68.el9 2024-01-19T15:44:44.642 INFO:teuthology.orchestra.run.smithi181.stdout:Architecture: x86_64 2024-01-19T15:44:44.642 INFO:teuthology.orchestra.run.smithi181.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-19T15:44:44.642 INFO:teuthology.orchestra.run.smithi181.stdout:Group : Unspecified 2024-01-19T15:44:44.642 INFO:teuthology.orchestra.run.smithi181.stdout:Size : 8264254 2024-01-19T15:44:44.642 INFO:teuthology.orchestra.run.smithi181.stdout:License : GPLv3+ 2024-01-19T15:44:44.642 INFO:teuthology.orchestra.run.smithi181.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-19T15:44:44.642 INFO:teuthology.orchestra.run.smithi181.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-19T15:44:44.642 INFO:teuthology.orchestra.run.smithi181.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-19T15:44:44.642 INFO:teuthology.orchestra.run.smithi181.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-19T15:44:44.642 INFO:teuthology.orchestra.run.smithi181.stdout:Packager : builder@centos.org 2024-01-19T15:44:44.642 INFO:teuthology.orchestra.run.smithi181.stdout:Vendor : CentOS 2024-01-19T15:44:44.643 INFO:teuthology.orchestra.run.smithi181.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-19T15:44:44.643 INFO:teuthology.orchestra.run.smithi181.stdout:Summary : Support tools for GRUB. 2024-01-19T15:44:44.643 INFO:teuthology.orchestra.run.smithi181.stdout:Description : 2024-01-19T15:44:44.643 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:44:44.643 INFO:teuthology.orchestra.run.smithi181.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-19T15:44:44.643 INFO:teuthology.orchestra.run.smithi181.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-19T15:44:44.643 INFO:teuthology.orchestra.run.smithi181.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-19T15:44:44.643 INFO:teuthology.orchestra.run.smithi181.stdout:hardware devices. 2024-01-19T15:44:44.643 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:44:44.643 INFO:teuthology.orchestra.run.smithi181.stdout:This subpackage provides tools for support of all platforms. 2024-01-19T15:44:44.644 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-19T15:44:44.644 INFO:teuthology.task.kernel:Updating grub on smithi181 to boot 5.14.0-409.el9.x86_64 2024-01-19T15:44:44.644 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-19T15:44:45.357 INFO:teuthology.orchestra.run.smithi181.stderr:Generating grub configuration file ... 2024-01-19T15:44:46.825 INFO:teuthology.orchestra.run.smithi181.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-19T15:44:46.862 INFO:teuthology.orchestra.run.smithi181.stderr:done 2024-01-19T15:44:46.864 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-01-19T15:44:46.908 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.prHbwcRk7A 2024-01-19T15:44:46.909 DEBUG:teuthology.orchestra.run.smithi181:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.prHbwcRk7A 2024-01-19T15:44:46.937 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.prHbwcRk7A 2024-01-19T15:44:47.077 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.prHbwcRk7A is 6KB 2024-01-19T15:44:47.125 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.prHbwcRk7A 2024-01-19T15:44:47.140 DEBUG:teuthology.orchestra.run.smithi181:> sudo /bin/ls /boot/loader/entries || true 2024-01-19T15:44:47.213 INFO:teuthology.orchestra.run.smithi181.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-19T15:44:47.213 INFO:teuthology.orchestra.run.smithi181.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-19T15:44:47.213 INFO:teuthology.orchestra.run.smithi181.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-19T15:44:47.213 INFO:teuthology.orchestra.run.smithi181.stdout:15949e3c88704300882482eddfd7baae-5.14.0-409.el9.x86_64.conf 2024-01-19T15:44:47.214 DEBUG:teuthology.orchestra.run.smithi181:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-409.el9.x86_64 2024-01-19T15:44:47.354 DEBUG:teuthology.orchestra.run.smithi181:> sudo shutdown -r now 2024-01-19T15:45:17.360 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-01-19T15:45:17.361 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-01-19T15:45:17.362 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T15:46:17.420 DEBUG:teuthology.orchestra.remote:timed out 2024-01-19T15:46:26.428 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-01-19T15:46:26.429 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T15:46:30.139 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-01-19T15:46:30.621 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-01-19T15:46:30.621 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-409.el9.x86_64"... 2024-01-19T15:46:30.621 DEBUG:teuthology.orchestra.run.smithi181:> uname -r 2024-01-19T15:46:30.636 INFO:teuthology.orchestra.run.smithi181.stdout:5.14.0-409.el9.x86_64 2024-01-19T15:46:30.636 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-409.el9.x86_64 vs 5.14.0-409.el9.x86_64 2024-01-19T15:46:30.636 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-19T15:46:30.636 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-19T15:46:31.637 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-01-19T15:46:31.638 DEBUG:teuthology.orchestra.run.smithi181:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-19T15:46:31.715 INFO:teuthology.orchestra.run.smithi181.stdout:ttyS1 2024-01-19T15:46:31.754 DEBUG:teuthology.parallel:result is None 2024-01-19T15:46:31.755 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-19T15:46:31.764 INFO:teuthology.task.internal:Creating test directory... 2024-01-19T15:46:31.764 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-19T15:46:31.779 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-19T15:46:31.787 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-19T15:46:31.795 INFO:teuthology.task.internal:Creating archive directory... 2024-01-19T15:46:31.795 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-19T15:46:31.840 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-19T15:46:31.846 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-19T15:46:31.847 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-19T15:46:31.907 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-19T15:46:31.921 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-19T15:46:31.922 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-19T15:46:31.931 INFO:teuthology.task.internal:Configuring sudo... 2024-01-19T15:46:31.931 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-19T15:46:31.990 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-19T15:46:31.999 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-19T15:46:31.999 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-19T15:46:32.045 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-19T15:46:32.128 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-19T15:46:32.179 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-19T15:46:32.260 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-19T15:46:32.302 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-01-19T15:46:32.303 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-19T15:46:32.371 DEBUG:teuthology.orchestra.run.smithi181:> sudo service rsyslog restart 2024-01-19T15:46:32.458 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-19T15:46:32.742 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-19T15:46:32.749 INFO:teuthology.task.internal:Starting timer... 2024-01-19T15:46:32.749 INFO:teuthology.run_tasks:Running task pcp... 2024-01-19T15:46:32.761 INFO:teuthology.run_tasks:Running task selinux... 2024-01-19T15:46:32.770 DEBUG:teuthology.orchestra.run.smithi181:> sudo service auditd rotate 2024-01-19T15:46:32.827 INFO:teuthology.orchestra.run.smithi181.stdout:Rotating logs: 2024-01-19T15:46:32.828 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-19T15:46:32.829 DEBUG:teuthology.orchestra.run.smithi181:> /usr/sbin/getenforce 2024-01-19T15:46:32.842 INFO:teuthology.orchestra.run.smithi181.stdout:Permissive 2024-01-19T15:46:32.842 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi181.front.sepia.ceph.com': 'permissive'} 2024-01-19T15:46:32.843 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-19T15:46:32.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-19T15:46:32.909 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-19T15:46:32.909 DEBUG:teuthology.orchestra.run.smithi181:> sudo /usr/sbin/setenforce permissive 2024-01-19T15:46:32.988 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-19T15:46:32.998 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-19T15:46:33.016 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-19T15:46:33.017 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi181.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-19T15:50:32.071 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi181.front.sepia.ceph.com')] 2024-01-19T15:50:32.072 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-01-19T15:50:32.073 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-19T15:50:32.164 DEBUG:teuthology.orchestra.run.smithi181:> true 2024-01-19T15:50:32.243 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi181.front.sepia.ceph.com' 2024-01-19T15:50:32.243 INFO:teuthology.run_tasks:Running task clock... 2024-01-19T15:50:32.254 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-19T15:50:32.255 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-19T15:50:32.255 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-19T15:50:32.315 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-19T15:50:32.332 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-19T15:50:32.361 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: ntpd: command not found 2024-01-19T15:50:32.374 INFO:teuthology.orchestra.run.smithi181.stdout:506 Cannot talk to daemon 2024-01-19T15:50:32.390 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-19T15:50:32.406 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-19T15:50:32.458 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-01-19T15:50:32.462 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-19T15:50:32.462 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-01-19T15:50:32.462 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T15:50:32.462 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T15:50:32.462 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T15:50:32.462 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T15:50:32.463 INFO:teuthology.run_tasks:Running task install... 2024-01-19T15:50:32.471 DEBUG:teuthology.task.install:project ceph 2024-01-19T15:50:32.471 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': 'bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4'}} 2024-01-19T15:50:32.471 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': 'bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4'} 2024-01-19T15:50:32.471 INFO:teuthology.task.install:Using flavor: crimson 2024-01-19T15:50:32.477 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-19T15:50:32.477 INFO:teuthology.task.install:extra packages: [] 2024-01-19T15:50:32.477 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4', 'tag': None, 'wait_for_package': False} 2024-01-19T15:50:32.478 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4 2024-01-19T15:50:32.619 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yingxin-crimson-save-conn-foreign-copy-8/bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4/centos/9/flavors/crimson/ 2024-01-19T15:50:32.619 INFO:teuthology.task.install.rpm:Package version is 19.0.0-767.gbdd6070c 2024-01-19T15:50:33.014 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yingxin-crimson-save-conn-foreign-copy-8/bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yingxin-crimson-save-conn-foreign-copy-8/bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yingxin-crimson-save-conn-foreign-copy-8/bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-19T15:50:33.014 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-01-19T15:50:33.014 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-19T15:50:33.043 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-19T15:50:33.043 DEBUG:teuthology.orchestra.run.smithi181:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-19T15:50:33.111 DEBUG:teuthology.orchestra.run.smithi181:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-19T15:50:33.191 DEBUG:teuthology.orchestra.run.smithi181:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-19T15:50:33.259 INFO:teuthology.orchestra.run.smithi181.stdout:check_obsoletes = 1 2024-01-19T15:50:33.260 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum clean all 2024-01-19T15:50:33.509 INFO:teuthology.orchestra.run.smithi181.stdout:73 files removed 2024-01-19T15:50:33.536 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-radosgw 2024-01-19T15:50:34.481 INFO:teuthology.orchestra.run.smithi181.stdout:ceph packages for x86_64 133 kB/s | 83 kB 00:00 2024-01-19T15:50:34.840 INFO:teuthology.orchestra.run.smithi181.stdout:ceph noarch packages 31 kB/s | 9.5 kB 00:00 2024-01-19T15:50:35.207 INFO:teuthology.orchestra.run.smithi181.stdout:ceph source packages 6.0 kB/s | 2.0 kB 00:00 2024-01-19T15:50:35.835 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS-9 - AppStream 11 MB/s | 6.4 MB 00:00 2024-01-19T15:50:37.201 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-01-19T15:50:38.784 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - BaseOS 6.4 MB/s | 8.0 MB 00:01 2024-01-19T15:50:43.060 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - AppStream 5.8 MB/s | 19 MB 00:03 2024-01-19T15:50:48.119 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 5.9 MB 00:01 2024-01-19T15:50:49.478 INFO:teuthology.orchestra.run.smithi181.stdout:CentOS Stream 9 - Extras packages 65 kB/s | 15 kB 00:00 2024-01-19T15:50:52.096 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 8.1 MB/s | 20 MB 00:02 2024-01-19T15:50:56.962 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-01-19T15:50:59.471 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:50:59.484 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:50:59.484 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-01-19T15:50:59.484 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:50:59.484 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-01-19T15:50:59.484 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 8.4 M 2024-01-19T15:50:59.485 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrading: 2024-01-19T15:50:59.485 INFO:teuthology.orchestra.run.smithi181.stdout: librados2 x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 3.1 M 2024-01-19T15:50:59.485 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1 x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 3.1 M 2024-01-19T15:50:59.485 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-01-19T15:50:59.485 INFO:teuthology.orchestra.run.smithi181.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-19T15:50:59.485 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 5.1 M 2024-01-19T15:50:59.485 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 18 M 2024-01-19T15:50:59.485 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 25 k 2024-01-19T15:50:59.485 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-19T15:50:59.485 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-19T15:50:59.485 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-19T15:50:59.485 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2 x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 752 k 2024-01-19T15:50:59.486 INFO:teuthology.orchestra.run.smithi181.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-19T15:50:59.486 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-19T15:50:59.486 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-19T15:50:59.486 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1 x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 496 k 2024-01-19T15:50:59.486 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-19T15:50:59.486 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2 x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 4.5 M 2024-01-19T15:50:59.486 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-19T15:50:59.486 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-19T15:50:59.486 INFO:teuthology.orchestra.run.smithi181.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-19T15:50:59.486 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 45 k 2024-01-19T15:50:59.487 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 130 k 2024-01-19T15:50:59.487 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 162 k 2024-01-19T15:50:59.487 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-19T15:50:59.487 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 320 k 2024-01-19T15:50:59.487 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 298 k 2024-01-19T15:50:59.487 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 99 k 2024-01-19T15:50:59.487 INFO:teuthology.orchestra.run.smithi181.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-19T15:50:59.487 INFO:teuthology.orchestra.run.smithi181.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-19T15:50:59.487 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:50:59.487 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-01-19T15:50:59.488 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:50:59.488 INFO:teuthology.orchestra.run.smithi181.stdout:Install 27 Packages 2024-01-19T15:50:59.488 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrade 2 Packages 2024-01-19T15:50:59.488 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:50:59.488 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 53 M 2024-01-19T15:50:59.488 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-01-19T15:51:04.655 INFO:teuthology.orchestra.run.smithi181.stdout:(1/29): ceph-common-19.0.0-767.gbdd6070c.el9.x8 3.4 MB/s | 18 MB 00:05 2024-01-19T15:51:04.747 INFO:teuthology.orchestra.run.smithi181.stdout:(2/29): ceph-selinux-19.0.0-767.gbdd6070c.el9.x 275 kB/s | 25 kB 00:00 2024-01-19T15:51:04.955 INFO:teuthology.orchestra.run.smithi181.stdout:(3/29): libcephfs2-19.0.0-767.gbdd6070c.el9.x86 3.5 MB/s | 752 kB 00:00 2024-01-19T15:51:05.097 INFO:teuthology.orchestra.run.smithi181.stdout:(4/29): libradosstriper1-19.0.0-767.gbdd6070c.e 3.4 MB/s | 496 kB 00:00 2024-01-19T15:51:06.006 INFO:teuthology.orchestra.run.smithi181.stdout:(5/29): librgw2-19.0.0-767.gbdd6070c.el9.x86_64 4.9 MB/s | 4.5 MB 00:00 2024-01-19T15:51:06.089 INFO:teuthology.orchestra.run.smithi181.stdout:(6/29): python3-ceph-argparse-19.0.0-767.gbdd60 543 kB/s | 45 kB 00:00 2024-01-19T15:51:06.181 INFO:teuthology.orchestra.run.smithi181.stdout:(7/29): python3-ceph-common-19.0.0-767.gbdd6070 1.4 MB/s | 130 kB 00:00 2024-01-19T15:51:06.282 INFO:teuthology.orchestra.run.smithi181.stdout:(8/29): python3-cephfs-19.0.0-767.gbdd6070c.el9 1.6 MB/s | 162 kB 00:00 2024-01-19T15:51:06.382 INFO:teuthology.orchestra.run.smithi181.stdout:(9/29): python3-rados-19.0.0-767.gbdd6070c.el9. 3.1 MB/s | 320 kB 00:00 2024-01-19T15:51:06.490 INFO:teuthology.orchestra.run.smithi181.stdout:(10/29): python3-rbd-19.0.0-767.gbdd6070c.el9.x 2.7 MB/s | 298 kB 00:00 2024-01-19T15:51:06.574 INFO:teuthology.orchestra.run.smithi181.stdout:(11/29): python3-rgw-19.0.0-767.gbdd6070c.el9.x 1.2 MB/s | 99 kB 00:00 2024-01-19T15:51:06.758 INFO:teuthology.orchestra.run.smithi181.stdout:(12/29): boost-program-options-1.75.0-8.el9.x86 578 kB/s | 106 kB 00:00 2024-01-19T15:51:06.808 INFO:teuthology.orchestra.run.smithi181.stdout:(13/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-01-19T15:51:06.841 INFO:teuthology.orchestra.run.smithi181.stdout:(14/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-01-19T15:51:06.908 INFO:teuthology.orchestra.run.smithi181.stdout:(15/29): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-01-19T15:51:06.951 INFO:teuthology.orchestra.run.smithi181.stdout:(16/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-01-19T15:51:06.984 INFO:teuthology.orchestra.run.smithi181.stdout:(17/29): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-01-19T15:51:07.418 INFO:teuthology.orchestra.run.smithi181.stdout:(18/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 713 kB/s | 309 kB 00:00 2024-01-19T15:51:09.052 INFO:teuthology.orchestra.run.smithi181.stdout:(19/29): ceph-base-19.0.0-767.gbdd6070c.el9.x86 554 kB/s | 5.1 MB 00:09 2024-01-19T15:51:09.193 INFO:teuthology.orchestra.run.smithi181.stdout:(20/29): libarrow-doc-9.0.0-10.el9.noarch.rpm 184 kB/s | 26 kB 00:00 2024-01-19T15:51:09.327 INFO:teuthology.orchestra.run.smithi181.stdout:(21/29): libarrow-9.0.0-10.el9.x86_64.rpm 2.3 MB/s | 4.4 MB 00:01 2024-01-19T15:51:09.419 INFO:teuthology.orchestra.run.smithi181.stdout:(22/29): liboath-2.6.7-2.el9.x86_64.rpm 216 kB/s | 49 kB 00:00 2024-01-19T15:51:09.444 INFO:teuthology.orchestra.run.smithi181.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 576 kB/s | 67 kB 00:00 2024-01-19T15:51:09.569 INFO:teuthology.orchestra.run.smithi181.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-01-19T15:51:09.678 INFO:teuthology.orchestra.run.smithi181.stdout:(25/29): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.2 MB/s | 838 kB 00:00 2024-01-19T15:51:09.995 INFO:teuthology.orchestra.run.smithi181.stdout:(26/29): ceph-radosgw-19.0.0-767.gbdd6070c.el9. 818 kB/s | 8.4 MB 00:10 2024-01-19T15:51:10.203 INFO:teuthology.orchestra.run.smithi181.stdout:(27/29): thrift-0.15.0-2.el9.x86_64.rpm 2.5 MB/s | 1.6 MB 00:00 2024-01-19T15:51:10.945 INFO:teuthology.orchestra.run.smithi181.stdout:(28/29): librbd1-19.0.0-767.gbdd6070c.el9.x86_6 3.2 MB/s | 3.1 MB 00:00 2024-01-19T15:51:12.587 INFO:teuthology.orchestra.run.smithi181.stdout:(29/29): librados2-19.0.0-767.gbdd6070c.el9.x86 1.1 MB/s | 3.1 MB 00:02 2024-01-19T15:51:12.590 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:51:12.590 INFO:teuthology.orchestra.run.smithi181.stdout:Total 4.0 MB/s | 53 MB 00:13 2024-01-19T15:51:12.591 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-01-19T15:51:12.626 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-01-19T15:51:12.626 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-01-19T15:51:13.069 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-01-19T15:51:13.069 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-01-19T15:51:14.416 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-01-19T15:51:14.768 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2024-01-19T15:51:14.914 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librados2-2:19.0.0-767.gbdd6070c.el9.x86_64 2/31 2024-01-19T15:51:14.988 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:19.0.0-767.gbdd6070c.el9.x86_64 2/31 2024-01-19T15:51:15.079 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rados-2:19.0.0-767.gbdd6070c.el9.x86_64 3/31 2024-01-19T15:51:15.107 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs2-2:19.0.0-767.gbdd6070c.el9.x86_64 4/31 2024-01-19T15:51:15.190 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs2-2:19.0.0-767.gbdd6070c.el9.x86_64 4/31 2024-01-19T15:51:15.219 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/31 2024-01-19T15:51:15.245 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 6/31 2024-01-19T15:51:15.274 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : liboath-2.6.7-2.el9.x86_64 7/31 2024-01-19T15:51:15.309 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-argparse-2:19.0.0-767.gbdd6070c.el9.x 8/31 2024-01-19T15:51:15.379 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cephfs-2:19.0.0-767.gbdd6070c.el9.x86_64 9/31 2024-01-19T15:51:15.402 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libradosstriper1-2:19.0.0-767.gbdd6070c.el9.x86_64 10/31 2024-01-19T15:51:15.474 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libradosstriper1-2:19.0.0-767.gbdd6070c.el9.x86_64 10/31 2024-01-19T15:51:15.509 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : thrift-0.15.0-2.el9.x86_64 11/31 2024-01-19T15:51:15.615 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2024-01-19T15:51:15.644 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 13/31 2024-01-19T15:51:15.684 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2024-01-19T15:51:15.724 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2024-01-19T15:51:15.753 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 16/31 2024-01-19T15:51:15.786 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2024-01-19T15:51:16.105 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2024-01-19T15:51:16.152 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librbd1-2:19.0.0-767.gbdd6070c.el9.x86_64 19/31 2024-01-19T15:51:16.243 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:19.0.0-767.gbdd6070c.el9.x86_64 19/31 2024-01-19T15:51:16.270 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rbd-2:19.0.0-767.gbdd6070c.el9.x86_64 20/31 2024-01-19T15:51:16.313 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2024-01-19T15:51:16.744 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 22/31 2024-01-19T15:51:16.769 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librgw2-2:19.0.0-767.gbdd6070c.el9.x86_64 23/31 2024-01-19T15:51:16.839 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librgw2-2:19.0.0-767.gbdd6070c.el9.x86_64 23/31 2024-01-19T15:51:16.906 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rgw-2:19.0.0-767.gbdd6070c.el9.x86_64 24/31 2024-01-19T15:51:16.929 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-common-2:19.0.0-767.gbdd6070c.el9.x86 25/31 2024-01-19T15:51:18.958 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:19.0.0-767.gbdd6070c.el9.x86_64 26/31 2024-01-19T15:51:19.000 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-common-2:19.0.0-767.gbdd6070c.el9.x86_64 26/31 2024-01-19T15:51:19.562 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:19.0.0-767.gbdd6070c.el9.x86_64 26/31 2024-01-19T15:51:19.612 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-base-2:19.0.0-767.gbdd6070c.el9.x86_64 27/31 2024-01-19T15:51:19.711 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-base-2:19.0.0-767.gbdd6070c.el9.x86_64 27/31 2024-01-19T15:51:19.711 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-19T15:51:19.711 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-19T15:51:19.712 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:51:19.755 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-selinux-2:19.0.0-767.gbdd6070c.el9.x86_64 28/31 2024-01-19T15:51:28.978 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-selinux-2:19.0.0-767.gbdd6070c.el9.x86_64 28/31 2024-01-19T15:51:28.978 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /sys 2024-01-19T15:51:28.978 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /proc 2024-01-19T15:51:28.978 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /mnt 2024-01-19T15:51:28.978 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /var/tmp 2024-01-19T15:51:28.978 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /home 2024-01-19T15:51:28.978 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /root 2024-01-19T15:51:28.978 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /tmp 2024-01-19T15:51:28.978 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:51:29.831 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-radosgw-2:19.0.0-767.gbdd6070c.el9.x86_64 29/31 2024-01-19T15:51:29.855 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-767.gbdd6070c.el9.x86_64 29/31 2024-01-19T15:51:29.855 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T15:51:29.856 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-19T15:51:29.856 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-19T15:51:29.856 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-19T15:51:29.856 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:51:29.857 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-19T15:51:29.943 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2024-01-19T15:51:29.943 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-19T15:51:31.077 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2024-01-19T15:51:31.077 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-base-2:19.0.0-767.gbdd6070c.el9.x86_64 1/31 2024-01-19T15:51:31.077 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-common-2:19.0.0-767.gbdd6070c.el9.x86_64 2/31 2024-01-19T15:51:31.077 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-radosgw-2:19.0.0-767.gbdd6070c.el9.x86_64 3/31 2024-01-19T15:51:31.077 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-selinux-2:19.0.0-767.gbdd6070c.el9.x86_64 4/31 2024-01-19T15:51:31.077 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs2-2:19.0.0-767.gbdd6070c.el9.x86_64 5/31 2024-01-19T15:51:31.077 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libradosstriper1-2:19.0.0-767.gbdd6070c.el9.x86_64 6/31 2024-01-19T15:51:31.078 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librgw2-2:19.0.0-767.gbdd6070c.el9.x86_64 7/31 2024-01-19T15:51:31.078 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-argparse-2:19.0.0-767.gbdd6070c.el9.x 8/31 2024-01-19T15:51:31.078 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-common-2:19.0.0-767.gbdd6070c.el9.x86 9/31 2024-01-19T15:51:31.078 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cephfs-2:19.0.0-767.gbdd6070c.el9.x86_64 10/31 2024-01-19T15:51:31.078 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rados-2:19.0.0-767.gbdd6070c.el9.x86_64 11/31 2024-01-19T15:51:31.078 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rbd-2:19.0.0-767.gbdd6070c.el9.x86_64 12/31 2024-01-19T15:51:31.078 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rgw-2:19.0.0-767.gbdd6070c.el9.x86_64 13/31 2024-01-19T15:51:31.078 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2024-01-19T15:51:31.078 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2024-01-19T15:51:31.078 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2024-01-19T15:51:31.078 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2024-01-19T15:51:31.078 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2024-01-19T15:51:31.078 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2024-01-19T15:51:31.078 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2024-01-19T15:51:31.079 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 21/31 2024-01-19T15:51:31.079 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 22/31 2024-01-19T15:51:31.079 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2024-01-19T15:51:31.079 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2024-01-19T15:51:31.079 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 25/31 2024-01-19T15:51:31.079 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2024-01-19T15:51:31.079 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2024-01-19T15:51:31.079 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:19.0.0-767.gbdd6070c.el9.x86_64 28/31 2024-01-19T15:51:31.079 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2024-01-19T15:51:31.080 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:19.0.0-767.gbdd6070c.el9.x86_64 30/31 2024-01-19T15:51:32.161 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2024-01-19T15:51:32.162 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:51:32.162 INFO:teuthology.orchestra.run.smithi181.stdout:Upgraded: 2024-01-19T15:51:32.162 INFO:teuthology.orchestra.run.smithi181.stdout: librados2-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:51:32.162 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:51:32.162 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-01-19T15:51:32.162 INFO:teuthology.orchestra.run.smithi181.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-19T15:51:32.162 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:51:32.162 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:51:32.162 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:51:32.162 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:51:32.162 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-19T15:51:32.163 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-19T15:51:32.163 INFO:teuthology.orchestra.run.smithi181.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-19T15:51:32.163 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:51:32.163 INFO:teuthology.orchestra.run.smithi181.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-19T15:51:32.163 INFO:teuthology.orchestra.run.smithi181.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-19T15:51:32.163 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-19T15:51:32.163 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:51:32.163 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-19T15:51:32.163 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:51:32.163 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-19T15:51:32.163 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-19T15:51:32.163 INFO:teuthology.orchestra.run.smithi181.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-19T15:51:32.163 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:51:32.163 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:51:32.163 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:51:32.164 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-19T15:51:32.164 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:51:32.164 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:51:32.164 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:51:32.164 INFO:teuthology.orchestra.run.smithi181.stdout: re2-1:20211101-3.el9.x86_64 2024-01-19T15:51:32.164 INFO:teuthology.orchestra.run.smithi181.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-19T15:51:32.164 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:51:32.164 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:51:32.495 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-test 2024-01-19T15:51:33.045 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:00:37 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:51:33.197 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:51:33.198 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:51:33.199 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-01-19T15:51:33.199 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:51:33.199 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-01-19T15:51:33.199 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 119 M 2024-01-19T15:51:33.199 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-01-19T15:51:33.199 INFO:teuthology.orchestra.run.smithi181.stdout: cryptopp x86_64 8.6.0-1.el9 epel 1.3 M 2024-01-19T15:51:33.199 INFO:teuthology.orchestra.run.smithi181.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-19T15:51:33.199 INFO:teuthology.orchestra.run.smithi181.stdout: libasan x86_64 11.4.1-3.el9 CentOS-AppStream 417 k 2024-01-19T15:51:33.199 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 158 k 2024-01-19T15:51:33.199 INFO:teuthology.orchestra.run.smithi181.stdout: libubsan x86_64 11.4.1-3.el9 CentOS-AppStream 186 k 2024-01-19T15:51:33.199 INFO:teuthology.orchestra.run.smithi181.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-19T15:51:33.199 INFO:teuthology.orchestra.run.smithi181.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-19T15:51:33.200 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-19T15:51:33.200 INFO:teuthology.orchestra.run.smithi181.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-01-19T15:51:33.200 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:51:33.200 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-01-19T15:51:33.200 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:51:33.200 INFO:teuthology.orchestra.run.smithi181.stdout:Install 10 Packages 2024-01-19T15:51:33.200 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:51:33.200 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 122 M 2024-01-19T15:51:33.200 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 1.1 G 2024-01-19T15:51:33.200 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-01-19T15:51:33.548 INFO:teuthology.orchestra.run.smithi181.stdout:(1/10): libasan-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 417 kB 00:00 2024-01-19T15:51:33.665 INFO:teuthology.orchestra.run.smithi181.stdout:(2/10): libubsan-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 186 kB 00:00 2024-01-19T15:51:33.715 INFO:teuthology.orchestra.run.smithi181.stdout:(3/10): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-01-19T15:51:33.749 INFO:teuthology.orchestra.run.smithi181.stdout:(4/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-01-19T15:51:33.774 INFO:teuthology.orchestra.run.smithi181.stdout:(5/10): libcephsqlite-19.0.0-767.gbdd6070c.el9. 396 kB/s | 158 kB 00:00 2024-01-19T15:51:33.799 INFO:teuthology.orchestra.run.smithi181.stdout:(6/10): jq-1.6-16.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2024-01-19T15:51:33.825 INFO:teuthology.orchestra.run.smithi181.stdout:(7/10): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-01-19T15:51:34.075 INFO:teuthology.orchestra.run.smithi181.stdout:(8/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 488 kB/s | 122 kB 00:00 2024-01-19T15:51:34.208 INFO:teuthology.orchestra.run.smithi181.stdout:(9/10): cryptopp-8.6.0-1.el9.x86_64.rpm 3.1 MB/s | 1.3 MB 00:00 2024-01-19T15:51:50.533 INFO:teuthology.orchestra.run.smithi181.stdout:(10/10): ceph-test-19.0.0-767.gbdd6070c.el9.x86 7.0 MB/s | 119 MB 00:17 2024-01-19T15:51:50.537 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:51:50.537 INFO:teuthology.orchestra.run.smithi181.stdout:Total 7.1 MB/s | 122 MB 00:17 2024-01-19T15:51:50.538 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-01-19T15:51:50.558 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-01-19T15:51:50.558 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-01-19T15:51:51.377 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-01-19T15:51:51.377 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-01-19T15:51:52.515 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-01-19T15:51:52.590 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2024-01-19T15:51:52.627 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cryptopp-8.6.0-1.el9.x86_64 2/10 2024-01-19T15:51:52.659 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/10 2024-01-19T15:51:52.691 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : jq-1.6-16.el9.x86_64 4/10 2024-01-19T15:51:52.742 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/10 2024-01-19T15:51:52.783 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 6/10 2024-01-19T15:51:52.835 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 7/10 2024-01-19T15:51:52.877 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libasan-11.4.1-3.el9.x86_64 8/10 2024-01-19T15:51:52.904 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephsqlite-2:19.0.0-767.gbdd6070c.el9.x86_64 9/10 2024-01-19T15:52:09.347 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephsqlite-2:19.0.0-767.gbdd6070c.el9.x86_64 9/10 2024-01-19T15:52:09.418 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-test-2:19.0.0-767.gbdd6070c.el9.x86_64 10/10 2024-01-19T15:52:17.702 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-test-2:19.0.0-767.gbdd6070c.el9.x86_64 10/10 2024-01-19T15:52:17.702 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-test-2:19.0.0-767.gbdd6070c.el9.x86_64 1/10 2024-01-19T15:52:17.702 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephsqlite-2:19.0.0-767.gbdd6070c.el9.x86_64 2/10 2024-01-19T15:52:17.702 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libasan-11.4.1-3.el9.x86_64 3/10 2024-01-19T15:52:17.702 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 4/10 2024-01-19T15:52:17.702 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/10 2024-01-19T15:52:17.702 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/10 2024-01-19T15:52:17.703 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : jq-1.6-16.el9.x86_64 7/10 2024-01-19T15:52:17.703 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 8/10 2024-01-19T15:52:17.703 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cryptopp-8.6.0-1.el9.x86_64 9/10 2024-01-19T15:52:18.114 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2024-01-19T15:52:18.114 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:52:18.114 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-01-19T15:52:18.114 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test-2:19.0.0-767.gbdd6070c.el9.x86_64 cryptopp-8.6.0-1.el9.x86_64 2024-01-19T15:52:18.114 INFO:teuthology.orchestra.run.smithi181.stdout: jq-1.6-16.el9.x86_64 libasan-11.4.1-3.el9.x86_64 2024-01-19T15:52:18.114 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite-2:19.0.0-767.gbdd6070c.el9.x86_64 libubsan-11.4.1-3.el9.x86_64 2024-01-19T15:52:18.114 INFO:teuthology.orchestra.run.smithi181.stdout: oniguruma-6.9.6-1.el9.5.x86_64 socat-1.7.4.1-5.el9.x86_64 2024-01-19T15:52:18.114 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet-1.6.1-20.el9.x86_64 yaml-cpp-0.6.3-5.el9.x86_64 2024-01-19T15:52:18.114 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:52:18.114 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:52:18.429 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph 2024-01-19T15:52:18.988 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:22 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:52:19.141 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:52:19.144 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================== 2024-01-19T15:52:19.144 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-01-19T15:52:19.144 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================== 2024-01-19T15:52:19.144 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-01-19T15:52:19.144 INFO:teuthology.orchestra.run.smithi181.stdout: ceph x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 6.6 k 2024-01-19T15:52:19.144 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-01-19T15:52:19.144 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 2.0 M 2024-01-19T15:52:19.144 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 1.5 M 2024-01-19T15:52:19.144 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core noarch 2:19.0.0-767.gbdd6070c.el9 ceph-noarch 245 k 2024-01-19T15:52:19.145 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 4.8 M 2024-01-19T15:52:19.145 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 117 M 2024-01-19T15:52:19.145 INFO:teuthology.orchestra.run.smithi181.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-01-19T15:52:19.145 INFO:teuthology.orchestra.run.smithi181.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-19T15:52:19.145 INFO:teuthology.orchestra.run.smithi181.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-19T15:52:19.145 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-19T15:52:19.145 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-19T15:52:19.145 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-19T15:52:19.145 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-19T15:52:19.145 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-19T15:52:19.145 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-19T15:52:19.146 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-19T15:52:19.146 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-19T15:52:19.146 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-19T15:52:19.146 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-19T15:52:19.146 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-19T15:52:19.146 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-19T15:52:19.146 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-19T15:52:19.146 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-19T15:52:19.146 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-19T15:52:19.146 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-19T15:52:19.146 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-19T15:52:19.146 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-19T15:52:19.146 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-19T15:52:19.146 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-19T15:52:19.147 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-19T15:52:19.147 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-19T15:52:19.147 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-19T15:52:19.147 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-19T15:52:19.147 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-19T15:52:19.147 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-19T15:52:19.147 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-19T15:52:19.147 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:52:19.147 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-01-19T15:52:19.147 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================== 2024-01-19T15:52:19.147 INFO:teuthology.orchestra.run.smithi181.stdout:Install 36 Packages 2024-01-19T15:52:19.147 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:52:19.148 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 130 M 2024-01-19T15:52:19.148 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 1.4 G 2024-01-19T15:52:19.148 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-01-19T15:52:19.636 INFO:teuthology.orchestra.run.smithi181.stdout:(1/36): ceph-19.0.0-767.gbdd6070c.el9.x86_64.rp 39 kB/s | 6.6 kB 00:00 2024-01-19T15:52:19.861 INFO:teuthology.orchestra.run.smithi181.stdout:(2/36): ceph-mgr-19.0.0-767.gbdd6070c.el9.x86_6 3.7 MB/s | 1.5 MB 00:00 2024-01-19T15:52:20.162 INFO:teuthology.orchestra.run.smithi181.stdout:(3/36): ceph-mds-19.0.0-767.gbdd6070c.el9.x86_6 2.9 MB/s | 2.0 MB 00:00 2024-01-19T15:52:20.245 INFO:teuthology.orchestra.run.smithi181.stdout:(4/36): ceph-mon-19.0.0-767.gbdd6070c.el9.x86_6 8.0 MB/s | 4.8 MB 00:00 2024-01-19T15:52:20.270 INFO:teuthology.orchestra.run.smithi181.stdout:(5/36): ceph-mgr-modules-core-19.0.0-767.gbdd60 2.2 MB/s | 245 kB 00:00 2024-01-19T15:52:20.396 INFO:teuthology.orchestra.run.smithi181.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-01-19T15:52:20.421 INFO:teuthology.orchestra.run.smithi181.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2024-01-19T15:52:20.438 INFO:teuthology.orchestra.run.smithi181.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 756 kB/s | 32 kB 00:00 2024-01-19T15:52:20.463 INFO:teuthology.orchestra.run.smithi181.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-01-19T15:52:20.488 INFO:teuthology.orchestra.run.smithi181.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 774 kB/s | 39 kB 00:00 2024-01-19T15:52:20.514 INFO:teuthology.orchestra.run.smithi181.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-01-19T15:52:20.606 INFO:teuthology.orchestra.run.smithi181.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 9.8 MB/s | 1.1 MB 00:00 2024-01-19T15:52:20.632 INFO:teuthology.orchestra.run.smithi181.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 877 kB/s | 103 kB 00:00 2024-01-19T15:52:20.657 INFO:teuthology.orchestra.run.smithi181.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-01-19T15:52:20.682 INFO:teuthology.orchestra.run.smithi181.stdout:(15/36): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-01-19T15:52:20.707 INFO:teuthology.orchestra.run.smithi181.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-01-19T15:52:20.766 INFO:teuthology.orchestra.run.smithi181.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-01-19T15:52:21.083 INFO:teuthology.orchestra.run.smithi181.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 137 kB/s | 43 kB 00:00 2024-01-19T15:52:21.141 INFO:teuthology.orchestra.run.smithi181.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 348 kB/s | 151 kB 00:00 2024-01-19T15:52:21.392 INFO:teuthology.orchestra.run.smithi181.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-01-19T15:52:21.417 INFO:teuthology.orchestra.run.smithi181.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 514 kB/s | 172 kB 00:00 2024-01-19T15:52:21.517 INFO:teuthology.orchestra.run.smithi181.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 85 kB/s | 11 kB 00:00 2024-01-19T15:52:21.543 INFO:teuthology.orchestra.run.smithi181.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 141 kB/s | 18 kB 00:00 2024-01-19T15:52:21.651 INFO:teuthology.orchestra.run.smithi181.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 173 kB/s | 23 kB 00:00 2024-01-19T15:52:21.668 INFO:teuthology.orchestra.run.smithi181.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 155 kB/s | 19 kB 00:00 2024-01-19T15:52:21.743 INFO:teuthology.orchestra.run.smithi181.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-01-19T15:52:21.786 INFO:teuthology.orchestra.run.smithi181.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 395 kB/s | 46 kB 00:00 2024-01-19T15:52:21.852 INFO:teuthology.orchestra.run.smithi181.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 726 kB/s | 79 kB 00:00 2024-01-19T15:52:21.961 INFO:teuthology.orchestra.run.smithi181.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 151 kB/s | 16 kB 00:00 2024-01-19T15:52:22.070 INFO:teuthology.orchestra.run.smithi181.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 959 kB/s | 272 kB 00:00 2024-01-19T15:52:22.095 INFO:teuthology.orchestra.run.smithi181.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 674 kB/s | 90 kB 00:00 2024-01-19T15:52:22.162 INFO:teuthology.orchestra.run.smithi181.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 390 kB/s | 36 kB 00:00 2024-01-19T15:52:22.362 INFO:teuthology.orchestra.run.smithi181.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 860 kB/s | 230 kB 00:00 2024-01-19T15:52:22.438 INFO:teuthology.orchestra.run.smithi181.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.5 MB/s | 427 kB 00:00 2024-01-19T15:52:22.463 INFO:teuthology.orchestra.run.smithi181.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-01-19T15:52:25.279 INFO:teuthology.orchestra.run.smithi181.stdout:(36/36): ceph-osd-19.0.0-767.gbdd6070c.el9.x86_ 22 MB/s | 117 MB 00:05 2024-01-19T15:52:25.284 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:52:25.284 INFO:teuthology.orchestra.run.smithi181.stdout:Total 21 MB/s | 130 MB 00:06 2024-01-19T15:52:25.303 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-01-19T15:52:25.345 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-01-19T15:52:25.345 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-01-19T15:52:26.337 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-01-19T15:52:26.338 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-01-19T15:52:28.238 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-01-19T15:52:28.298 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-01-19T15:52:28.331 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-01-19T15:52:28.368 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-01-19T15:52:28.439 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-01-19T15:52:28.483 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-01-19T15:52:28.516 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-01-19T15:52:28.616 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-01-19T15:52:28.695 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-01-19T15:52:28.736 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-01-19T15:52:28.808 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-01-19T15:52:28.862 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-01-19T15:52:28.900 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2024-01-19T15:52:28.949 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-01-19T15:52:29.003 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-01-19T15:52:29.192 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-01-19T15:52:29.246 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-01-19T15:52:29.282 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-01-19T15:52:29.319 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-01-19T15:52:29.349 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-01-19T15:52:29.425 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-01-19T15:52:29.459 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-19T15:52:29.493 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-01-19T15:52:29.533 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-01-19T15:52:29.568 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-01-19T15:52:29.611 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-01-19T15:52:29.699 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-01-19T15:52:29.744 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-01-19T15:52:29.817 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-01-19T15:52:29.955 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-01-19T15:52:30.079 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-19T15:52:30.239 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-767.gbdd6070c.el9.n 31/36 2024-01-19T15:52:30.266 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-2:19.0.0-767.gbdd6070c.el9.x86_64 32/36 2024-01-19T15:52:30.291 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-2:19.0.0-767.gbdd6070c.el9.x86_64 32/36 2024-01-19T15:52:30.292 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T15:52:30.292 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-19T15:52:30.292 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-19T15:52:30.292 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-19T15:52:30.292 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:52:48.599 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-osd-2:19.0.0-767.gbdd6070c.el9.x86_64 33/36 2024-01-19T15:52:48.624 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-osd-2:19.0.0-767.gbdd6070c.el9.x86_64 33/36 2024-01-19T15:52:48.624 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T15:52:48.624 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-19T15:52:48.624 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-19T15:52:48.625 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-19T15:52:48.625 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:52:49.085 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mon-2:19.0.0-767.gbdd6070c.el9.x86_64 34/36 2024-01-19T15:52:49.109 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mon-2:19.0.0-767.gbdd6070c.el9.x86_64 34/36 2024-01-19T15:52:49.109 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T15:52:49.109 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-19T15:52:49.109 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-19T15:52:49.110 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-19T15:52:49.110 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:52:49.312 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mds-2:19.0.0-767.gbdd6070c.el9.x86_64 35/36 2024-01-19T15:52:49.333 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mds-2:19.0.0-767.gbdd6070c.el9.x86_64 35/36 2024-01-19T15:52:49.333 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T15:52:49.333 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-19T15:52:49.333 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-19T15:52:49.333 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-19T15:52:49.333 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:52:49.400 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-2:19.0.0-767.gbdd6070c.el9.x86_64 36/36 2024-01-19T15:52:59.149 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-2:19.0.0-767.gbdd6070c.el9.x86_64 36/36 2024-01-19T15:52:59.149 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-2:19.0.0-767.gbdd6070c.el9.x86_64 1/36 2024-01-19T15:52:59.149 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mds-2:19.0.0-767.gbdd6070c.el9.x86_64 2/36 2024-01-19T15:52:59.149 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-2:19.0.0-767.gbdd6070c.el9.x86_64 3/36 2024-01-19T15:52:59.149 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mon-2:19.0.0-767.gbdd6070c.el9.x86_64 4/36 2024-01-19T15:52:59.149 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-osd-2:19.0.0-767.gbdd6070c.el9.x86_64 5/36 2024-01-19T15:52:59.149 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-767.gbdd6070c.el9.n 6/36 2024-01-19T15:52:59.149 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-01-19T15:52:59.149 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-01-19T15:52:59.150 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-01-19T15:52:59.150 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-01-19T15:52:59.150 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-01-19T15:52:59.150 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-01-19T15:52:59.150 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-01-19T15:52:59.150 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-01-19T15:52:59.150 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-01-19T15:52:59.150 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2024-01-19T15:52:59.150 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 17/36 2024-01-19T15:52:59.150 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-01-19T15:52:59.150 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-01-19T15:52:59.150 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-01-19T15:52:59.150 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-01-19T15:52:59.150 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-01-19T15:52:59.151 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-01-19T15:52:59.151 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-01-19T15:52:59.151 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-01-19T15:52:59.151 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-01-19T15:52:59.151 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-01-19T15:52:59.151 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-01-19T15:52:59.152 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-01-19T15:52:59.152 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-01-19T15:52:59.152 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-01-19T15:52:59.152 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-01-19T15:52:59.152 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-01-19T15:52:59.152 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-01-19T15:52:59.152 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-01-19T15:53:00.154 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-01-19T15:53:00.154 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:00.154 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-01-19T15:53:00.154 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:53:00.155 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:53:00.155 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:53:00.155 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core-2:19.0.0-767.gbdd6070c.el9.noarch 2024-01-19T15:53:00.155 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:53:00.155 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:53:00.155 INFO:teuthology.orchestra.run.smithi181.stdout: lua-5.4.4-4.el9.x86_64 2024-01-19T15:53:00.155 INFO:teuthology.orchestra.run.smithi181.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-19T15:53:00.156 INFO:teuthology.orchestra.run.smithi181.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-19T15:53:00.156 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-19T15:53:00.156 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-19T15:53:00.156 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-19T15:53:00.156 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-19T15:53:00.156 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-19T15:53:00.156 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-19T15:53:00.157 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-19T15:53:00.157 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-19T15:53:00.157 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-19T15:53:00.157 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-19T15:53:00.157 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-19T15:53:00.157 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-19T15:53:00.158 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-19T15:53:00.158 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-19T15:53:00.158 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-19T15:53:00.158 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ply-3.11-14.el9.noarch 2024-01-19T15:53:00.158 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-19T15:53:00.158 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-19T15:53:00.158 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-19T15:53:00.158 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-19T15:53:00.159 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-19T15:53:00.159 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-19T15:53:00.159 INFO:teuthology.orchestra.run.smithi181.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-19T15:53:00.159 INFO:teuthology.orchestra.run.smithi181.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-19T15:53:00.159 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-19T15:53:00.159 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-19T15:53:00.159 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-19T15:53:00.160 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:00.160 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:53:00.404 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-base 2024-01-19T15:53:00.976 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:04 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:53:01.077 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-base-2:19.0.0-767.gbdd6070c.el9.x86_64 is already installed. 2024-01-19T15:53:01.127 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:53:01.128 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-01-19T15:53:01.128 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:53:01.202 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install cephadm 2024-01-19T15:53:01.770 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:05 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:53:01.921 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:53:01.922 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:01.922 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-01-19T15:53:01.922 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:01.922 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-01-19T15:53:01.922 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm noarch 2:19.0.0-767.gbdd6070c.el9 ceph-noarch 742 k 2024-01-19T15:53:01.922 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:01.923 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-01-19T15:53:01.923 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:01.923 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-01-19T15:53:01.923 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:01.923 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 742 k 2024-01-19T15:53:01.923 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 748 k 2024-01-19T15:53:01.923 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-01-19T15:53:02.468 INFO:teuthology.orchestra.run.smithi181.stdout:cephadm-19.0.0-767.gbdd6070c.el9.noarch.rpm 1.3 MB/s | 742 kB 00:00 2024-01-19T15:53:02.469 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:53:02.469 INFO:teuthology.orchestra.run.smithi181.stdout:Total 1.3 MB/s | 742 kB 00:00 2024-01-19T15:53:02.469 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-01-19T15:53:02.474 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-01-19T15:53:02.474 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-01-19T15:53:02.480 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-01-19T15:53:02.480 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-01-19T15:53:02.554 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-01-19T15:53:02.998 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:19.0.0-767.gbdd6070c.el9.noarch 1/1 2024-01-19T15:53:03.081 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephadm-2:19.0.0-767.gbdd6070c.el9.noarch 1/1 2024-01-19T15:53:03.491 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:19.0.0-767.gbdd6070c.el9.noarch 1/1 2024-01-19T15:53:03.846 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephadm-2:19.0.0-767.gbdd6070c.el9.noarch 1/1 2024-01-19T15:53:03.846 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:03.846 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-01-19T15:53:03.846 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm-2:19.0.0-767.gbdd6070c.el9.noarch 2024-01-19T15:53:03.846 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:03.847 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:53:04.046 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-immutable-object-cache 2024-01-19T15:53:04.619 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:08 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:53:04.774 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:53:04.774 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:04.774 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-01-19T15:53:04.775 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:04.775 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-01-19T15:53:04.775 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 147 k 2024-01-19T15:53:04.775 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:04.775 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-01-19T15:53:04.775 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:04.775 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-01-19T15:53:04.775 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:04.775 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 147 k 2024-01-19T15:53:04.775 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 481 k 2024-01-19T15:53:04.775 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-01-19T15:53:05.024 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-immutable-object-cache-19.0.0-767.gbdd6070 593 kB/s | 147 kB 00:00 2024-01-19T15:53:05.025 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:53:05.025 INFO:teuthology.orchestra.run.smithi181.stdout:Total 589 kB/s | 147 kB 00:00 2024-01-19T15:53:05.025 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-01-19T15:53:05.032 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-01-19T15:53:05.032 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-01-19T15:53:05.083 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-01-19T15:53:05.083 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-01-19T15:53:05.218 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-01-19T15:53:05.275 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-767.gbdd6070c.e 1/1 2024-01-19T15:53:05.300 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-767.gbdd6070c.e 1/1 2024-01-19T15:53:05.300 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T15:53:05.300 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-19T15:53:05.300 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:06.355 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-767.gbdd6070c.e 1/1 2024-01-19T15:53:06.355 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:06.355 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-01-19T15:53:06.355 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:53:06.355 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:06.355 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:53:06.576 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr 2024-01-19T15:53:07.148 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:11 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:53:07.248 INFO:teuthology.orchestra.run.smithi181.stdout:Package ceph-mgr-2:19.0.0-767.gbdd6070c.el9.x86_64 is already installed. 2024-01-19T15:53:07.298 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:53:07.299 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-01-19T15:53:07.299 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:53:07.372 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-dashboard 2024-01-19T15:53:07.932 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:11 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:53:08.081 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:53:08.083 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:08.083 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-01-19T15:53:08.083 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:08.084 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-01-19T15:53:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard noarch 2:19.0.0-767.gbdd6070c.el9 ceph-noarch 1.7 M 2024-01-19T15:53:08.084 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-01-19T15:53:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards noarch 2:19.0.0-767.gbdd6070c.el9 ceph-noarch 28 k 2024-01-19T15:53:08.084 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts noarch 2:19.0.0-767.gbdd6070c.el9 ceph-noarch 15 k 2024-01-19T15:53:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-19T15:53:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-19T15:53:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:08.085 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-01-19T15:53:08.085 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:08.085 INFO:teuthology.orchestra.run.smithi181.stdout:Install 5 Packages 2024-01-19T15:53:08.085 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:08.086 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 2.0 M 2024-01-19T15:53:08.086 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 8.5 M 2024-01-19T15:53:08.086 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-01-19T15:53:08.637 INFO:teuthology.orchestra.run.smithi181.stdout:(1/5): ceph-grafana-dashboards-19.0.0-767.gbdd6 147 kB/s | 28 kB 00:00 2024-01-19T15:53:08.737 INFO:teuthology.orchestra.run.smithi181.stdout:(2/5): ceph-prometheus-alerts-19.0.0-767.gbdd60 52 kB/s | 15 kB 00:00 2024-01-19T15:53:08.754 INFO:teuthology.orchestra.run.smithi181.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 263 kB/s | 31 kB 00:00 2024-01-19T15:53:08.845 INFO:teuthology.orchestra.run.smithi181.stdout:(4/5): ceph-mgr-dashboard-19.0.0-767.gbdd6070c. 4.3 MB/s | 1.7 MB 00:00 2024-01-19T15:53:08.912 INFO:teuthology.orchestra.run.smithi181.stdout:(5/5): python3-routes-2.5.1-5.el9.noarch.rpm 1.0 MB/s | 188 kB 00:00 2024-01-19T15:53:08.914 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:53:08.914 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2024-01-19T15:53:08.915 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-01-19T15:53:08.921 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-01-19T15:53:08.922 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-01-19T15:53:08.984 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-01-19T15:53:08.984 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-01-19T15:53:09.226 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-01-19T15:53:09.301 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-01-19T15:53:09.342 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-01-19T15:53:09.372 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-767.gbdd6070c.el9.no 3/5 2024-01-19T15:53:09.606 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-767.gbdd6070c.el9.n 4/5 2024-01-19T15:53:09.640 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-767.gbdd6070c.el9.noarch 5/5 2024-01-19T15:53:10.170 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-767.gbdd6070c.el9.noarch 5/5 2024-01-19T15:53:10.170 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-767.gbdd6070c.el9.n 1/5 2024-01-19T15:53:10.170 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-767.gbdd6070c.el9.noarch 2/5 2024-01-19T15:53:10.170 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-767.gbdd6070c.el9.no 3/5 2024-01-19T15:53:10.171 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-01-19T15:53:10.479 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-01-19T15:53:10.479 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:10.479 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-01-19T15:53:10.479 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards-2:19.0.0-767.gbdd6070c.el9.noarch 2024-01-19T15:53:10.479 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard-2:19.0.0-767.gbdd6070c.el9.noarch 2024-01-19T15:53:10.479 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts-2:19.0.0-767.gbdd6070c.el9.noarch 2024-01-19T15:53:10.479 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-19T15:53:10.480 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-19T15:53:10.480 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:10.480 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:53:10.700 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-19T15:53:11.260 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:15 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:53:11.412 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:53:11.413 INFO:teuthology.orchestra.run.smithi181.stdout:=========================================================================================== 2024-01-19T15:53:11.413 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-01-19T15:53:11.413 INFO:teuthology.orchestra.run.smithi181.stdout:=========================================================================================== 2024-01-19T15:53:11.413 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-01-19T15:53:11.414 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-767.gbdd6070c.el9 ceph-noarch 7.4 M 2024-01-19T15:53:11.414 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-01-19T15:53:11.414 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-19T15:53:11.414 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-19T15:53:11.414 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-19T15:53:11.414 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-19T15:53:11.414 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-19T15:53:11.414 INFO:teuthology.orchestra.run.smithi181.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-19T15:53:11.414 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-19T15:53:11.414 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-19T15:53:11.414 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-19T15:53:11.414 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-19T15:53:11.414 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-19T15:53:11.415 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:11.415 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-01-19T15:53:11.415 INFO:teuthology.orchestra.run.smithi181.stdout:=========================================================================================== 2024-01-19T15:53:11.415 INFO:teuthology.orchestra.run.smithi181.stdout:Install 12 Packages 2024-01-19T15:53:11.415 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:11.415 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 37 M 2024-01-19T15:53:11.415 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 209 M 2024-01-19T15:53:11.415 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-01-19T15:53:11.602 INFO:teuthology.orchestra.run.smithi181.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 221 kB/s | 32 kB 00:00 2024-01-19T15:53:11.635 INFO:teuthology.orchestra.run.smithi181.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 536 kB/s | 17 kB 00:00 2024-01-19T15:53:11.677 INFO:teuthology.orchestra.run.smithi181.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 842 kB/s | 34 kB 00:00 2024-01-19T15:53:11.736 INFO:teuthology.orchestra.run.smithi181.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-01-19T15:53:11.903 INFO:teuthology.orchestra.run.smithi181.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-01-19T15:53:11.978 INFO:teuthology.orchestra.run.smithi181.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-01-19T15:53:12.003 INFO:teuthology.orchestra.run.smithi181.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-01-19T15:53:12.162 INFO:teuthology.orchestra.run.smithi181.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 5.0 MB/s | 802 kB 00:00 2024-01-19T15:53:12.337 INFO:teuthology.orchestra.run.smithi181.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-01-19T15:53:12.371 INFO:teuthology.orchestra.run.smithi181.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 941 kB/s | 194 kB 00:00 2024-01-19T15:53:12.538 INFO:teuthology.orchestra.run.smithi181.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-01-19T15:53:12.980 INFO:teuthology.orchestra.run.smithi181.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-7 4.9 MB/s | 7.4 MB 00:01 2024-01-19T15:53:12.982 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:53:12.982 INFO:teuthology.orchestra.run.smithi181.stdout:Total 24 MB/s | 37 MB 00:01 2024-01-19T15:53:12.995 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-01-19T15:53:13.020 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-01-19T15:53:13.020 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-01-19T15:53:13.413 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-01-19T15:53:13.413 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-01-19T15:53:14.132 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-01-19T15:53:14.211 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-01-19T15:53:14.242 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-01-19T15:53:14.334 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-01-19T15:53:14.370 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-01-19T15:53:14.558 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-19T15:53:14.586 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-19T15:53:14.663 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-01-19T15:53:15.306 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-01-19T15:53:15.423 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-01-19T15:53:16.649 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-01-19T15:53:17.676 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-01-19T15:53:17.703 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-767.gbdd607 12/12 2024-01-19T15:53:20.181 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-767.gbdd607 12/12 2024-01-19T15:53:20.181 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-767.gbdd607 1/12 2024-01-19T15:53:20.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-01-19T15:53:20.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-01-19T15:53:20.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-01-19T15:53:20.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-01-19T15:53:20.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-01-19T15:53:20.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-01-19T15:53:20.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-01-19T15:53:20.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-01-19T15:53:20.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-01-19T15:53:20.182 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-01-19T15:53:20.615 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-01-19T15:53:20.616 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:20.616 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-01-19T15:53:20.616 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local-2:19.0.0-767.gbdd6070c.el9.noarch 2024-01-19T15:53:20.616 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-19T15:53:20.616 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-19T15:53:20.616 INFO:teuthology.orchestra.run.smithi181.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-19T15:53:20.616 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-19T15:53:20.616 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-19T15:53:20.616 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-19T15:53:20.616 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-19T15:53:20.616 INFO:teuthology.orchestra.run.smithi181.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-19T15:53:20.616 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-19T15:53:20.617 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-19T15:53:20.617 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-19T15:53:20.617 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:20.617 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:53:21.059 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-rook 2024-01-19T15:53:21.629 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:25 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:53:21.782 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:53:21.783 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-01-19T15:53:21.783 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-01-19T15:53:21.783 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-01-19T15:53:21.783 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-01-19T15:53:21.783 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook noarch 2:19.0.0-767.gbdd6070c.el9 ceph-noarch 51 k 2024-01-19T15:53:21.783 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-01-19T15:53:21.783 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-19T15:53:21.783 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-19T15:53:21.783 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth noarch 1:2.26.0-1.el9 epel 202 k 2024-01-19T15:53:21.783 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-19T15:53:21.783 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-19T15:53:21.784 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-19T15:53:21.784 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-19T15:53:21.784 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-19T15:53:21.784 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-19T15:53:21.784 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-19T15:53:21.784 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-19T15:53:21.784 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-19T15:53:21.784 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:21.784 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-01-19T15:53:21.784 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-01-19T15:53:21.784 INFO:teuthology.orchestra.run.smithi181.stdout:Install 13 Packages 2024-01-19T15:53:21.784 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:21.785 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 2.0 M 2024-01-19T15:53:21.785 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 26 M 2024-01-19T15:53:21.785 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-01-19T15:53:22.278 INFO:teuthology.orchestra.run.smithi181.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 185 kB/s | 24 kB 00:00 2024-01-19T15:53:22.303 INFO:teuthology.orchestra.run.smithi181.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 115 kB/s | 18 kB 00:00 2024-01-19T15:53:22.362 INFO:teuthology.orchestra.run.smithi181.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 132 kB 00:00 2024-01-19T15:53:22.387 INFO:teuthology.orchestra.run.smithi181.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-01-19T15:53:22.412 INFO:teuthology.orchestra.run.smithi181.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-01-19T15:53:22.438 INFO:teuthology.orchestra.run.smithi181.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 854 kB/s | 43 kB 00:00 2024-01-19T15:53:22.463 INFO:teuthology.orchestra.run.smithi181.stdout:(7/13): ceph-mgr-rook-19.0.0-767.gbdd6070c.el9. 160 kB/s | 51 kB 00:00 2024-01-19T15:53:22.688 INFO:teuthology.orchestra.run.smithi181.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2024-01-19T15:53:22.713 INFO:teuthology.orchestra.run.smithi181.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 107 kB/s | 32 kB 00:00 2024-01-19T15:53:22.938 INFO:teuthology.orchestra.run.smithi181.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 262 kB/s | 59 kB 00:00 2024-01-19T15:53:22.964 INFO:teuthology.orchestra.run.smithi181.stdout:(11/13): python3-google-auth-2.26.0-1.el9.noarc 403 kB/s | 202 kB 00:00 2024-01-19T15:53:23.131 INFO:teuthology.orchestra.run.smithi181.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 467 kB/s | 90 kB 00:00 2024-01-19T15:53:23.231 INFO:teuthology.orchestra.run.smithi181.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2024-01-19T15:53:23.233 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:53:23.233 INFO:teuthology.orchestra.run.smithi181.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2024-01-19T15:53:23.234 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-01-19T15:53:23.247 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-01-19T15:53:23.247 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-01-19T15:53:23.349 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-01-19T15:53:23.349 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-01-19T15:53:23.869 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-01-19T15:53:24.021 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-01-19T15:53:24.082 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-01-19T15:53:24.146 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-01-19T15:53:24.183 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-01-19T15:53:24.225 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-01-19T15:53:24.327 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-01-19T15:53:24.435 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-google-auth-1:2.26.0-1.el9.noarch 7/13 2024-01-19T15:53:24.492 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-01-19T15:53:25.217 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-01-19T15:53:25.273 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-01-19T15:53:25.307 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-01-19T15:53:25.363 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-01-19T15:53:25.385 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-rook-2:19.0.0-767.gbdd6070c.el9.noarch 13/13 2024-01-19T15:53:26.180 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-767.gbdd6070c.el9.noarch 13/13 2024-01-19T15:53:26.180 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-rook-2:19.0.0-767.gbdd6070c.el9.noarch 1/13 2024-01-19T15:53:26.180 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-01-19T15:53:26.180 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-01-19T15:53:26.180 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-01-19T15:53:26.180 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-01-19T15:53:26.181 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-01-19T15:53:26.181 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-01-19T15:53:26.181 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-01-19T15:53:26.181 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-01-19T15:53:26.181 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-google-auth-1:2.26.0-1.el9.noarch 10/13 2024-01-19T15:53:26.181 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-01-19T15:53:26.181 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-01-19T15:53:26.570 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-01-19T15:53:26.570 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:26.570 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-01-19T15:53:26.570 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook-2:19.0.0-767.gbdd6070c.el9.noarch 2024-01-19T15:53:26.570 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-19T15:53:26.570 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-19T15:53:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth-1:2.26.0-1.el9.noarch 2024-01-19T15:53:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-19T15:53:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-19T15:53:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-19T15:53:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-19T15:53:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-19T15:53:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-19T15:53:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-19T15:53:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-19T15:53:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-19T15:53:26.571 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:26.571 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:53:26.839 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-mgr-cephadm 2024-01-19T15:53:27.402 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:31 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:53:27.556 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:53:27.557 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-01-19T15:53:27.557 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-01-19T15:53:27.557 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-01-19T15:53:27.557 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-01-19T15:53:27.558 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm noarch 2:19.0.0-767.gbdd6070c.el9 ceph-noarch 133 k 2024-01-19T15:53:27.558 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-01-19T15:53:27.558 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-19T15:53:27.558 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-19T15:53:27.558 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-19T15:53:27.558 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-19T15:53:27.558 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-19T15:53:27.558 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:27.558 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-01-19T15:53:27.558 INFO:teuthology.orchestra.run.smithi181.stdout:======================================================================================= 2024-01-19T15:53:27.558 INFO:teuthology.orchestra.run.smithi181.stdout:Install 6 Packages 2024-01-19T15:53:27.558 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:27.559 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 6.8 M 2024-01-19T15:53:27.559 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 33 M 2024-01-19T15:53:27.559 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-01-19T15:53:27.916 INFO:teuthology.orchestra.run.smithi181.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.3 MB/s | 228 kB 00:00 2024-01-19T15:53:28.058 INFO:teuthology.orchestra.run.smithi181.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-01-19T15:53:28.175 INFO:teuthology.orchestra.run.smithi181.stdout:(3/6): ceph-mgr-cephadm-19.0.0-767.gbdd6070c.el 307 kB/s | 133 kB 00:00 2024-01-19T15:53:29.226 INFO:teuthology.orchestra.run.smithi181.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 49 kB/s | 58 kB 00:01 2024-01-19T15:53:29.301 INFO:teuthology.orchestra.run.smithi181.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 45 kB/s | 51 kB 00:01 2024-01-19T15:53:29.726 INFO:teuthology.orchestra.run.smithi181.stdout:(6/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 303 kB/s | 548 kB 00:01 2024-01-19T15:53:29.731 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:53:29.731 INFO:teuthology.orchestra.run.smithi181.stdout:Total 3.1 MB/s | 6.8 MB 00:02 2024-01-19T15:53:29.732 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-01-19T15:53:29.742 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-01-19T15:53:29.742 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-01-19T15:53:29.836 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-01-19T15:53:29.837 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-01-19T15:53:30.134 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-01-19T15:53:30.289 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-01-19T15:53:30.344 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2024-01-19T15:53:30.867 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-01-19T15:53:30.945 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-01-19T15:53:31.006 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2024-01-19T15:53:31.031 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-767.gbdd6070c.el9.noarch 6/6 2024-01-19T15:53:31.830 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-767.gbdd6070c.el9.noarch 6/6 2024-01-19T15:53:31.830 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-767.gbdd6070c.el9.noarch 1/6 2024-01-19T15:53:31.831 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-01-19T15:53:31.831 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2024-01-19T15:53:31.831 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2024-01-19T15:53:31.831 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-01-19T15:53:32.173 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-01-19T15:53:32.173 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:32.174 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-01-19T15:53:32.174 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm-2:19.0.0-767.gbdd6070c.el9.noarch 2024-01-19T15:53:32.174 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-19T15:53:32.174 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-19T15:53:32.174 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-19T15:53:32.174 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-19T15:53:32.174 INFO:teuthology.orchestra.run.smithi181.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-19T15:53:32.174 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:32.174 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:53:32.442 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-fuse 2024-01-19T15:53:33.000 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:37 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:53:33.152 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:53:33.152 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:33.152 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-01-19T15:53:33.152 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:33.152 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-01-19T15:53:33.152 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 858 k 2024-01-19T15:53:33.152 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-01-19T15:53:33.153 INFO:teuthology.orchestra.run.smithi181.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-19T15:53:33.153 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:33.153 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-01-19T15:53:33.153 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:33.153 INFO:teuthology.orchestra.run.smithi181.stdout:Install 2 Packages 2024-01-19T15:53:33.153 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:33.154 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 937 k 2024-01-19T15:53:33.154 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 3.1 M 2024-01-19T15:53:33.154 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-01-19T15:53:33.286 INFO:teuthology.orchestra.run.smithi181.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 597 kB/s | 79 kB 00:00 2024-01-19T15:53:33.744 INFO:teuthology.orchestra.run.smithi181.stdout:(2/2): ceph-fuse-19.0.0-767.gbdd6070c.el9.x86_6 1.4 MB/s | 858 kB 00:00 2024-01-19T15:53:33.745 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:53:33.746 INFO:teuthology.orchestra.run.smithi181.stdout:Total 1.5 MB/s | 937 kB 00:00 2024-01-19T15:53:33.746 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-01-19T15:53:33.766 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-01-19T15:53:33.766 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-01-19T15:53:33.841 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-01-19T15:53:33.841 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-01-19T15:53:34.036 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-01-19T15:53:34.164 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-01-19T15:53:34.236 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-fuse-2:19.0.0-767.gbdd6070c.el9.x86_64 2/2 2024-01-19T15:53:34.959 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-fuse-2:19.0.0-767.gbdd6070c.el9.x86_64 2/2 2024-01-19T15:53:34.960 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-fuse-2:19.0.0-767.gbdd6070c.el9.x86_64 1/2 2024-01-19T15:53:35.286 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-01-19T15:53:35.286 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:35.286 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-01-19T15:53:35.286 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse-2:19.0.0-767.gbdd6070c.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-01-19T15:53:35.286 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:35.286 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:53:35.545 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-volume 2024-01-19T15:53:36.114 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:40 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:53:36.265 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:53:36.266 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:36.266 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-01-19T15:53:36.266 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:36.266 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-01-19T15:53:36.266 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume noarch 2:19.0.0-767.gbdd6070c.el9 ceph-noarch 262 k 2024-01-19T15:53:36.267 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:36.267 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-01-19T15:53:36.267 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:36.267 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-01-19T15:53:36.267 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:36.267 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 262 k 2024-01-19T15:53:36.267 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 1.2 M 2024-01-19T15:53:36.267 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-01-19T15:53:36.515 INFO:teuthology.orchestra.run.smithi181.stdout:ceph-volume-19.0.0-767.gbdd6070c.el9.noarch.rpm 1.0 MB/s | 262 kB 00:00 2024-01-19T15:53:36.516 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:53:36.516 INFO:teuthology.orchestra.run.smithi181.stdout:Total 1.0 MB/s | 262 kB 00:00 2024-01-19T15:53:36.516 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-01-19T15:53:36.524 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-01-19T15:53:36.524 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-01-19T15:53:36.561 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-01-19T15:53:36.562 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-01-19T15:53:36.771 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-01-19T15:53:36.842 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-volume-2:19.0.0-767.gbdd6070c.el9.noarch 1/1 2024-01-19T15:53:36.857 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-volume-2:19.0.0-767.gbdd6070c.el9.noarch 1/1 2024-01-19T15:53:36.858 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T15:53:36.858 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-19T15:53:36.858 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:37.883 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-volume-2:19.0.0-767.gbdd6070c.el9.noarch 1/1 2024-01-19T15:53:37.883 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:37.883 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-01-19T15:53:37.883 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume-2:19.0.0-767.gbdd6070c.el9.noarch 2024-01-19T15:53:37.883 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:37.883 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:53:38.160 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados-devel 2024-01-19T15:53:38.733 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:42 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:53:38.888 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:53:38.889 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:38.889 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-01-19T15:53:38.889 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:38.889 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-01-19T15:53:38.889 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 127 k 2024-01-19T15:53:38.889 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:38.889 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-01-19T15:53:38.889 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:38.889 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-01-19T15:53:38.890 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:38.890 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 127 k 2024-01-19T15:53:38.890 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 457 k 2024-01-19T15:53:38.890 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-01-19T15:53:39.104 INFO:teuthology.orchestra.run.smithi181.stdout:librados-devel-19.0.0-767.gbdd6070c.el9.x86_64. 592 kB/s | 127 kB 00:00 2024-01-19T15:53:39.104 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:53:39.105 INFO:teuthology.orchestra.run.smithi181.stdout:Total 588 kB/s | 127 kB 00:00 2024-01-19T15:53:39.105 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-01-19T15:53:39.110 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-01-19T15:53:39.110 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-01-19T15:53:39.165 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-01-19T15:53:39.165 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-01-19T15:53:39.328 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-01-19T15:53:39.437 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librados-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:53:39.925 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:53:40.257 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:53:40.257 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:40.257 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-01-19T15:53:40.257 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:53:40.257 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:40.257 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:53:40.490 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs2 2024-01-19T15:53:41.065 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:45 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:53:41.170 INFO:teuthology.orchestra.run.smithi181.stdout:Package libcephfs2-2:19.0.0-767.gbdd6070c.el9.x86_64 is already installed. 2024-01-19T15:53:41.222 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:53:41.223 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-01-19T15:53:41.223 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:53:41.291 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install libcephfs-devel 2024-01-19T15:53:41.851 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:45 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:53:42.003 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:53:42.004 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:42.004 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-01-19T15:53:42.004 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:42.004 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-01-19T15:53:42.004 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 31 k 2024-01-19T15:53:42.004 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:42.004 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-01-19T15:53:42.004 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:42.004 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-01-19T15:53:42.004 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:42.004 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 31 k 2024-01-19T15:53:42.004 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 140 k 2024-01-19T15:53:42.004 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-01-19T15:53:42.274 INFO:teuthology.orchestra.run.smithi181.stdout:libcephfs-devel-19.0.0-767.gbdd6070c.el9.x86_64 116 kB/s | 31 kB 00:00 2024-01-19T15:53:42.275 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:53:42.275 INFO:teuthology.orchestra.run.smithi181.stdout:Total 115 kB/s | 31 kB 00:00 2024-01-19T15:53:42.275 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-01-19T15:53:42.278 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-01-19T15:53:42.279 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-01-19T15:53:42.303 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-01-19T15:53:42.304 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-01-19T15:53:42.395 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-01-19T15:53:42.522 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:53:42.910 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:53:43.200 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:53:43.200 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:43.200 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-01-19T15:53:43.200 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:53:43.200 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:43.200 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:53:43.383 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librados2 2024-01-19T15:53:43.945 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:47 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:53:44.046 INFO:teuthology.orchestra.run.smithi181.stdout:Package librados2-2:19.0.0-767.gbdd6070c.el9.x86_64 is already installed. 2024-01-19T15:53:44.097 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:53:44.098 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-01-19T15:53:44.099 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:53:44.169 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install librbd1 2024-01-19T15:53:44.728 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:48 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:53:44.828 INFO:teuthology.orchestra.run.smithi181.stdout:Package librbd1-2:19.0.0-767.gbdd6070c.el9.x86_64 is already installed. 2024-01-19T15:53:44.880 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:53:44.881 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-01-19T15:53:44.881 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:53:44.948 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rados 2024-01-19T15:53:45.508 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:49 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:53:45.609 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rados-2:19.0.0-767.gbdd6070c.el9.x86_64 is already installed. 2024-01-19T15:53:45.659 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:53:45.660 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-01-19T15:53:45.660 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:53:45.734 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rgw 2024-01-19T15:53:46.299 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:50 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:53:46.400 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rgw-2:19.0.0-767.gbdd6070c.el9.x86_64 is already installed. 2024-01-19T15:53:46.453 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:53:46.453 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-01-19T15:53:46.454 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:53:46.553 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-cephfs 2024-01-19T15:53:47.134 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:51 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:53:47.247 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-cephfs-2:19.0.0-767.gbdd6070c.el9.x86_64 is already installed. 2024-01-19T15:53:47.300 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:53:47.301 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-01-19T15:53:47.301 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:53:47.376 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-rbd 2024-01-19T15:53:47.935 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:51 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:53:48.035 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-rbd-2:19.0.0-767.gbdd6070c.el9.x86_64 is already installed. 2024-01-19T15:53:48.087 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:53:48.088 INFO:teuthology.orchestra.run.smithi181.stdout:Nothing to do. 2024-01-19T15:53:48.088 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:53:48.167 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-fuse 2024-01-19T15:53:48.743 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:52 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:53:48.903 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:53:48.904 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:48.904 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-01-19T15:53:48.904 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:48.905 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-01-19T15:53:48.905 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 86 k 2024-01-19T15:53:48.905 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:48.905 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-01-19T15:53:48.905 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:48.905 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-01-19T15:53:48.905 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:48.905 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 86 k 2024-01-19T15:53:48.905 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 256 k 2024-01-19T15:53:48.905 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-01-19T15:53:49.519 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-fuse-19.0.0-767.gbdd6070c.el9.x86_64.rpm 140 kB/s | 86 kB 00:00 2024-01-19T15:53:49.520 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:53:49.520 INFO:teuthology.orchestra.run.smithi181.stdout:Total 139 kB/s | 86 kB 00:00 2024-01-19T15:53:49.521 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-01-19T15:53:49.528 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-01-19T15:53:49.528 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-01-19T15:53:49.584 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-01-19T15:53:49.585 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-01-19T15:53:49.732 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-01-19T15:53:49.834 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-fuse-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:53:50.296 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-fuse-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:53:50.607 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-fuse-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:53:50.607 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:50.607 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-01-19T15:53:50.607 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:53:50.607 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:50.607 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:53:50.811 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-mirror 2024-01-19T15:53:51.387 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:55 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:53:51.542 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:53:51.543 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:51.543 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repo Size 2024-01-19T15:53:51.543 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:51.543 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-01-19T15:53:51.543 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 3.0 M 2024-01-19T15:53:51.543 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:51.543 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-01-19T15:53:51.543 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:51.543 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-01-19T15:53:51.544 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:51.544 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 3.0 M 2024-01-19T15:53:51.544 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 14 M 2024-01-19T15:53:51.544 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-01-19T15:53:52.772 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-mirror-19.0.0-767.gbdd6070c.el9.x86_64.rpm 2.5 MB/s | 3.0 MB 00:01 2024-01-19T15:53:52.773 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:53:52.773 INFO:teuthology.orchestra.run.smithi181.stdout:Total 2.5 MB/s | 3.0 MB 00:01 2024-01-19T15:53:52.773 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-01-19T15:53:52.783 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-01-19T15:53:52.784 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-01-19T15:53:52.857 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-01-19T15:53:52.857 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-01-19T15:53:53.287 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-01-19T15:53:53.356 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-mirror-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:53:53.380 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-mirror-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:53:53.380 INFO:teuthology.orchestra.run.smithi181.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-19T15:53:53.381 INFO:teuthology.orchestra.run.smithi181.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-19T15:53:53.381 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-19T15:53:53.381 INFO:teuthology.orchestra.run.smithi181.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-19T15:53:53.381 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:54.608 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-mirror-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:53:54.608 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:54.608 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-01-19T15:53:54.608 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:53:54.608 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:54.608 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:53:54.885 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install rbd-nbd 2024-01-19T15:53:55.459 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:02:59 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:53:55.610 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:53:55.610 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:55.610 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2024-01-19T15:53:55.610 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:55.610 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-01-19T15:53:55.610 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 169 k 2024-01-19T15:53:55.611 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:55.611 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-01-19T15:53:55.611 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:53:55.611 INFO:teuthology.orchestra.run.smithi181.stdout:Install 1 Package 2024-01-19T15:53:55.611 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:55.611 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 169 k 2024-01-19T15:53:55.611 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 537 k 2024-01-19T15:53:55.612 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-01-19T15:53:55.976 INFO:teuthology.orchestra.run.smithi181.stdout:rbd-nbd-19.0.0-767.gbdd6070c.el9.x86_64.rpm 465 kB/s | 169 kB 00:00 2024-01-19T15:53:55.976 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:53:55.976 INFO:teuthology.orchestra.run.smithi181.stdout:Total 463 kB/s | 169 kB 00:00 2024-01-19T15:53:55.976 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-01-19T15:53:55.984 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-01-19T15:53:55.984 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-01-19T15:53:56.039 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-01-19T15:53:56.039 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-01-19T15:53:56.186 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-01-19T15:53:56.290 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-nbd-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:53:56.754 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-nbd-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:53:57.055 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-nbd-2:19.0.0-767.gbdd6070c.el9.x86_64 1/1 2024-01-19T15:53:57.055 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:57.056 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-01-19T15:53:57.056 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:53:57.056 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:57.056 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:53:57.320 DEBUG:teuthology.parallel:result is None 2024-01-19T15:53:57.321 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4 2024-01-19T15:53:57.464 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-19T15:53:57.484 INFO:teuthology.orchestra.run.smithi181.stdout:19.0.0-767.gbdd6070c.el9 2024-01-19T15:53:57.485 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-767.gbdd6070c.el9 2024-01-19T15:53:57.485 INFO:teuthology.task.install:The correct ceph version 19.0.0-767.gbdd6070c is installed. 2024-01-19T15:53:57.486 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-19T15:53:57.487 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-01-19T15:53:57.487 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-19T15:53:57.552 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-19T15:53:57.553 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-01-19T15:53:57.553 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/daemon-helper 2024-01-19T15:53:57.619 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-19T15:53:57.685 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-19T15:53:57.686 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-01-19T15:53:57.686 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-19T15:53:57.753 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-19T15:53:57.819 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-19T15:53:57.820 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-01-19T15:53:57.820 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/stdin-killer 2024-01-19T15:53:57.887 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-19T15:53:57.953 INFO:teuthology.run_tasks:Running task ceph... 2024-01-19T15:53:58.038 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-19T15:53:58.038 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 777 /var/log/ceph 2024-01-19T15:53:58.066 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-19T15:53:58.066 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-19T15:53:58.133 INFO:tasks.ceph:Creating extra log directories... 2024-01-19T15:53:58.134 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-19T15:53:58.208 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-19T15:53:58.208 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson osd obc lru size': 10, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-19T15:53:58.209 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yingxin-2024-01-19_07:32:18-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-8-distro-default-smithi/7521663', 'branch': 'wip-yingxin-crimson-save-conn-foreign-copy-8', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read}', 'email': 'yingxin.cheng@intel.com', 'first_in_suite': False, 'job_id': '7521663', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yingxin-2024-01-19_07:32:18-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-8-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yingxin-crimson-save-conn-foreign-copy-8'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson osd obc lru size': 10, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': 'bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4'}}, 'workunit': {'branch': 'wip-yingxin-crimson-save-conn-foreign-copy-8', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4'}}, 'owner': 'scheduled_yingxin@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'sha1': 'bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'wip-yingxin-crimson-save-conn-foreign-copy-8', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4', 'targets': {'smithi181.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNoUEJRLjb2j1PKdTbNzHU5CltHfFrjhnEZwFVvTj+KEbI/zu9HTGnalyR4eiHbcaehdOG9zc+189044MkncjLI='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson osd obc lru size': 10, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'bdd6070c73b3e82e684d10c1e8bc493fbaf64eb4', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randread'], 'norandommap': True, 'op_size': [4194304], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2]}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'cd45576300487d997e5a85abed65500b9f5d143b', 'timestamp': '2024-01-19_07:32:18', 'tube': 'smithi', 'user': 'yingxin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-19T15:53:58.209 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-19T15:53:58.255 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m0777 -- /var/run/ceph 2024-01-19T15:53:58.322 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-01-19T15:53:58.322 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/scratch_devs of=/dev/stdout 2024-01-19T15:53:58.377 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-19T15:53:58.377 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_1 2024-01-19T15:53:58.431 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-19T15:53:58.431 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-19T15:53:58.432 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-01-19T15:53:58.432 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T15:53:58.432 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-01-19T15:53:58.432 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-01-19 15:53:53.654248445 +0000 2024-01-19T15:53:58.432 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-01-19 15:50:09.776025403 +0000 2024-01-19T15:53:58.432 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-01-19 15:50:09.776025403 +0000 2024-01-19T15:53:58.432 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-01-19 15:50:09.776025403 +0000 2024-01-19T15:53:58.432 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-19T15:53:58.497 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-01-19T15:53:58.497 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-01-19T15:53:58.497 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000195325 s, 2.6 MB/s 2024-01-19T15:53:58.498 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-19T15:53:58.553 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_2 2024-01-19T15:53:58.609 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-19T15:53:58.609 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-19T15:53:58.609 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-01-19T15:53:58.609 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T15:53:58.609 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-01-19T15:53:58.609 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-01-19 15:53:53.654248445 +0000 2024-01-19T15:53:58.609 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-01-19 15:50:10.122018714 +0000 2024-01-19T15:53:58.609 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-01-19 15:50:10.122018714 +0000 2024-01-19T15:53:58.609 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-01-19 15:50:10.122018714 +0000 2024-01-19T15:53:58.609 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-19T15:53:58.674 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-01-19T15:53:58.674 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-01-19T15:53:58.674 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000183591 s, 2.8 MB/s 2024-01-19T15:53:58.675 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-19T15:53:58.731 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_3 2024-01-19T15:53:58.786 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-19T15:53:58.786 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-19T15:53:58.786 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-01-19T15:53:58.786 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T15:53:58.786 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-01-19T15:53:58.786 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-01-19 15:53:53.655248425 +0000 2024-01-19T15:53:58.786 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-01-19 15:50:10.467012044 +0000 2024-01-19T15:53:58.786 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-01-19 15:50:10.467012044 +0000 2024-01-19T15:53:58.786 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-01-19 15:50:10.467012044 +0000 2024-01-19T15:53:58.787 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-19T15:53:58.851 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-01-19T15:53:58.851 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-01-19T15:53:58.851 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000224552 s, 2.3 MB/s 2024-01-19T15:53:58.852 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-19T15:53:58.908 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_4 2024-01-19T15:53:58.963 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-19T15:53:58.963 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-19T15:53:58.963 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-01-19T15:53:58.963 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-19T15:53:58.963 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2024-01-19T15:53:58.964 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2024-01-19 15:53:53.655248425 +0000 2024-01-19T15:53:58.964 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2024-01-19 15:50:10.817005278 +0000 2024-01-19T15:53:58.964 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2024-01-19 15:50:10.817005278 +0000 2024-01-19T15:53:58.964 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: 2024-01-19 15:50:10.817005278 +0000 2024-01-19T15:53:58.964 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-19T15:53:59.028 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2024-01-19T15:53:59.028 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2024-01-19T15:53:59.028 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000213812 s, 2.4 MB/s 2024-01-19T15:53:59.029 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-19T15:53:59.084 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-19T15:53:59.084 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi181.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-01-19T15:53:59.084 INFO:tasks.ceph:Generating config... 2024-01-19T15:53:59.085 INFO:tasks.ceph:[osd] debug monc = 20 2024-01-19T15:53:59.086 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-01-19T15:53:59.086 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-01-19T15:53:59.086 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-01-19T15:53:59.086 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-01-19T15:53:59.086 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-01-19T15:53:59.086 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-01-19T15:53:59.086 INFO:tasks.ceph:[osd] crimson seastar smp = 3 2024-01-19T15:53:59.086 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-01-19T15:53:59.086 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-01-19T15:53:59.086 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-01-19T15:53:59.086 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-01-19T15:53:59.086 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-01-19T15:53:59.086 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-01-19T15:53:59.086 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-01-19T15:53:59.087 INFO:tasks.ceph:[osd] err to stderr = True 2024-01-19T15:53:59.087 INFO:tasks.ceph:[osd] log flush on exit = True 2024-01-19T15:53:59.087 INFO:tasks.ceph:[osd] log to file = False 2024-01-19T15:53:59.087 INFO:tasks.ceph:[osd] log to stderr = True 2024-01-19T15:53:59.087 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-01-19T15:53:59.087 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-01-19T15:53:59.087 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-01-19T15:53:59.087 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-01-19T15:53:59.087 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-01-19T15:53:59.087 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-01-19T15:53:59.087 INFO:tasks.ceph:[global] auth client required = none 2024-01-19T15:53:59.087 INFO:tasks.ceph:[global] auth cluster required = none 2024-01-19T15:53:59.087 INFO:tasks.ceph:[global] auth service required = none 2024-01-19T15:53:59.087 INFO:tasks.ceph:[global] auth supported = none 2024-01-19T15:53:59.088 INFO:tasks.ceph:[global] debug asok = 0/0 2024-01-19T15:53:59.088 INFO:tasks.ceph:[global] debug auth = 0/0 2024-01-19T15:53:59.088 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-01-19T15:53:59.088 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-01-19T15:53:59.088 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-01-19T15:53:59.088 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-01-19T15:53:59.088 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-01-19T15:53:59.088 INFO:tasks.ceph:[global] debug client = 0/0 2024-01-19T15:53:59.088 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-01-19T15:53:59.088 INFO:tasks.ceph:[global] debug context = 0/0 2024-01-19T15:53:59.088 INFO:tasks.ceph:[global] debug crush = 0/0 2024-01-19T15:53:59.088 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-01-19T15:53:59.088 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-01-19T15:53:59.088 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-01-19T15:53:59.089 INFO:tasks.ceph:[global] debug filer = 0/0 2024-01-19T15:53:59.089 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-01-19T15:53:59.089 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-01-19T15:53:59.089 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-01-19T15:53:59.089 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-01-19T15:53:59.089 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-01-19T15:53:59.089 INFO:tasks.ceph:[global] debug journal = 0/0 2024-01-19T15:53:59.089 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-01-19T15:53:59.089 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-01-19T15:53:59.089 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-01-19T15:53:59.089 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-01-19T15:53:59.089 INFO:tasks.ceph:[global] debug mds = 0/0 2024-01-19T15:53:59.089 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-01-19T15:53:59.089 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-01-19T15:53:59.089 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-01-19T15:53:59.090 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-01-19T15:53:59.090 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-01-19T15:53:59.090 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-01-19T15:53:59.090 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-01-19T15:53:59.090 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-01-19T15:53:59.090 INFO:tasks.ceph:[global] debug mon = 0/0 2024-01-19T15:53:59.090 INFO:tasks.ceph:[global] debug monc = 0/0 2024-01-19T15:53:59.090 INFO:tasks.ceph:[global] debug ms = 0/0 2024-01-19T15:53:59.090 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-01-19T15:53:59.090 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-01-19T15:53:59.090 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-01-19T15:53:59.090 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-01-19T15:53:59.090 INFO:tasks.ceph:[global] debug osd = 0/0 2024-01-19T15:53:59.090 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-01-19T15:53:59.090 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-01-19T15:53:59.090 INFO:tasks.ceph:[global] debug rados = 0/0 2024-01-19T15:53:59.091 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-01-19T15:53:59.091 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-01-19T15:53:59.091 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-01-19T15:53:59.091 INFO:tasks.ceph:[global] debug refs = 0/0 2024-01-19T15:53:59.091 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-01-19T15:53:59.091 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-01-19T15:53:59.091 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-01-19T15:53:59.091 INFO:tasks.ceph:[global] debug striper = 0/0 2024-01-19T15:53:59.091 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-01-19T15:53:59.091 INFO:tasks.ceph:[global] debug timer = 0/0 2024-01-19T15:53:59.091 INFO:tasks.ceph:[global] debug tp = 0/0 2024-01-19T15:53:59.091 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-01-19T15:53:59.091 INFO:tasks.ceph:[global] ms client mode = crc 2024-01-19T15:53:59.091 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-01-19T15:53:59.091 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-01-19T15:53:59.092 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-01-19T15:53:59.092 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-01-19T15:53:59.092 INFO:tasks.ceph:[global] ms service mode = crc 2024-01-19T15:53:59.092 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-19T15:53:59.092 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-19T15:53:59.092 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-01-19T15:53:59.092 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-01-19T15:53:59.092 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-01-19T15:53:59.092 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-01-19T15:53:59.092 INFO:tasks.ceph:Setting up mon.a... 2024-01-19T15:53:59.092 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-19T15:53:59.180 INFO:teuthology.orchestra.run.smithi181.stdout:creating /etc/ceph/ceph.keyring 2024-01-19T15:53:59.183 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-19T15:53:59.233 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-19T15:53:59.259 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.181')] 2024-01-19T15:53:59.259 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.181', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson osd obc lru size': 10, 'crimson seastar smp': 3, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2024-01-19T15:53:59.260 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-01-19T15:53:59.260 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-19T15:53:59.314 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.181 --print /home/ubuntu/cephtest/ceph.monmap 2024-01-19T15:53:59.399 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:53:59.407+0000 7f7e26e32e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:59.399 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:53:59.407+0000 7f7e26e32e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:59.399 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:53:59.408+0000 7f7e26e32e80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:53:59.401 INFO:teuthology.orchestra.run.smithi181.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-19T15:53:59.401 INFO:teuthology.orchestra.run.smithi181.stdout:monmaptool: generated fsid ced042d8-398d-4631-a7e9-4f6702eeddab 2024-01-19T15:53:59.401 INFO:teuthology.orchestra.run.smithi181.stdout:setting min_mon_release = quincy 2024-01-19T15:53:59.401 INFO:teuthology.orchestra.run.smithi181.stdout:epoch 0 2024-01-19T15:53:59.401 INFO:teuthology.orchestra.run.smithi181.stdout:fsid ced042d8-398d-4631-a7e9-4f6702eeddab 2024-01-19T15:53:59.401 INFO:teuthology.orchestra.run.smithi181.stdout:last_changed 2024-01-19T15:53:59.409076+0000 2024-01-19T15:53:59.401 INFO:teuthology.orchestra.run.smithi181.stdout:created 2024-01-19T15:53:59.409076+0000 2024-01-19T15:53:59.401 INFO:teuthology.orchestra.run.smithi181.stdout:min_mon_release 17 (quincy) 2024-01-19T15:53:59.401 INFO:teuthology.orchestra.run.smithi181.stdout:election_strategy: 1 2024-01-19T15:53:59.401 INFO:teuthology.orchestra.run.smithi181.stdout:0: [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon.a 2024-01-19T15:53:59.401 INFO:teuthology.orchestra.run.smithi181.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-01-19T15:53:59.402 DEBUG:teuthology.orchestra.run.smithi181:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-19T15:53:59.418 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ced042d8-398d-4631-a7e9-4f6702eeddab... 2024-01-19T15:53:59.419 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-19T15:53:59.509 INFO:teuthology.orchestra.run.smithi181.stdout:[global] 2024-01-19T15:53:59.509 INFO:teuthology.orchestra.run.smithi181.stdout: chdir = "" 2024-01-19T15:53:59.509 INFO:teuthology.orchestra.run.smithi181.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-19T15:53:59.509 INFO:teuthology.orchestra.run.smithi181.stdout: auth supported = none 2024-01-19T15:53:59.510 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.510 INFO:teuthology.orchestra.run.smithi181.stdout: filestore xattr use omap = true 2024-01-19T15:53:59.510 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.510 INFO:teuthology.orchestra.run.smithi181.stdout: mon clock drift allowed = 1.000 2024-01-19T15:53:59.510 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.510 INFO:teuthology.orchestra.run.smithi181.stdout: osd crush chooseleaf type = 0 2024-01-19T15:53:59.510 INFO:teuthology.orchestra.run.smithi181.stdout: auth debug = true 2024-01-19T15:53:59.510 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.510 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on old message = true 2024-01-19T15:53:59.510 INFO:teuthology.orchestra.run.smithi181.stdout: ms die on bug = true 2024-01-19T15:53:59.510 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.510 INFO:teuthology.orchestra.run.smithi181.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-19T15:53:59.510 INFO:teuthology.orchestra.run.smithi181.stdout: mon pg warn max object skew = 0 2024-01-19T15:53:59.510 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.510 INFO:teuthology.orchestra.run.smithi181.stdout: # disable pg_autoscaler by default for new pools 2024-01-19T15:53:59.511 INFO:teuthology.orchestra.run.smithi181.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-19T15:53:59.511 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.511 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default size = 2 2024-01-19T15:53:59.511 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.511 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow primary affinity = true 2024-01-19T15:53:59.511 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd allow pg remap = true 2024-01-19T15:53:59.511 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on legacy crush tunables = false 2024-01-19T15:53:59.511 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on crush straw calc version zero = false 2024-01-19T15:53:59.511 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on no sortbitwise = false 2024-01-19T15:53:59.511 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on osd down out interval zero = false 2024-01-19T15:53:59.511 INFO:teuthology.orchestra.run.smithi181.stdout: mon warn on too few osds = false 2024-01-19T15:53:59.511 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-19T15:53:59.511 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-19T15:53:59.511 INFO:teuthology.orchestra.run.smithi181.stdout: mon_allow_pool_size_one = true 2024-01-19T15:53:59.511 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.512 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-19T15:53:59.512 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.512 INFO:teuthology.orchestra.run.smithi181.stdout: osd default data pool replay window = 5 2024-01-19T15:53:59.512 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.512 INFO:teuthology.orchestra.run.smithi181.stdout: mon allow pool delete = true 2024-01-19T15:53:59.512 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.512 INFO:teuthology.orchestra.run.smithi181.stdout: mon cluster log file level = debug 2024-01-19T15:53:59.512 INFO:teuthology.orchestra.run.smithi181.stdout: debug asserts on shutdown = true 2024-01-19T15:53:59.512 INFO:teuthology.orchestra.run.smithi181.stdout: mon health detail to clog = false 2024-01-19T15:53:59.512 INFO:teuthology.orchestra.run.smithi181.stdout: mon host = 172.21.15.181 2024-01-19T15:53:59.512 INFO:teuthology.orchestra.run.smithi181.stdout: auth client required = none 2024-01-19T15:53:59.512 INFO:teuthology.orchestra.run.smithi181.stdout: auth cluster required = none 2024-01-19T15:53:59.512 INFO:teuthology.orchestra.run.smithi181.stdout: auth service required = none 2024-01-19T15:53:59.512 INFO:teuthology.orchestra.run.smithi181.stdout: debug asok = 0/0 2024-01-19T15:53:59.513 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 0/0 2024-01-19T15:53:59.513 INFO:teuthology.orchestra.run.smithi181.stdout: debug bdev = 0/0 2024-01-19T15:53:59.513 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 0/0 2024-01-19T15:53:59.513 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 0/0 2024-01-19T15:53:59.513 INFO:teuthology.orchestra.run.smithi181.stdout: debug buffer = 0/0 2024-01-19T15:53:59.513 INFO:teuthology.orchestra.run.smithi181.stdout: debug civetweb = 0/0 2024-01-19T15:53:59.513 INFO:teuthology.orchestra.run.smithi181.stdout: debug client = 0/0 2024-01-19T15:53:59.513 INFO:teuthology.orchestra.run.smithi181.stdout: debug compressor = 0/0 2024-01-19T15:53:59.513 INFO:teuthology.orchestra.run.smithi181.stdout: debug context = 0/0 2024-01-19T15:53:59.513 INFO:teuthology.orchestra.run.smithi181.stdout: debug crush = 0/0 2024-01-19T15:53:59.513 INFO:teuthology.orchestra.run.smithi181.stdout: debug crypto = 0/0 2024-01-19T15:53:59.513 INFO:teuthology.orchestra.run.smithi181.stdout: debug dpdk = 0/0 2024-01-19T15:53:59.513 INFO:teuthology.orchestra.run.smithi181.stdout: debug eventtrace = 0/0 2024-01-19T15:53:59.513 INFO:teuthology.orchestra.run.smithi181.stdout: debug filer = 0/0 2024-01-19T15:53:59.514 INFO:teuthology.orchestra.run.smithi181.stdout: debug filestore = 0/0 2024-01-19T15:53:59.514 INFO:teuthology.orchestra.run.smithi181.stdout: debug finisher = 0/0 2024-01-19T15:53:59.514 INFO:teuthology.orchestra.run.smithi181.stdout: debug fuse = 0/0 2024-01-19T15:53:59.514 INFO:teuthology.orchestra.run.smithi181.stdout: debug heartbeatmap = 0/0 2024-01-19T15:53:59.514 INFO:teuthology.orchestra.run.smithi181.stdout: debug javaclient = 0/0 2024-01-19T15:53:59.514 INFO:teuthology.orchestra.run.smithi181.stdout: debug journal = 0/0 2024-01-19T15:53:59.514 INFO:teuthology.orchestra.run.smithi181.stdout: debug journaler = 0/0 2024-01-19T15:53:59.514 INFO:teuthology.orchestra.run.smithi181.stdout: debug kstore = 0/0 2024-01-19T15:53:59.514 INFO:teuthology.orchestra.run.smithi181.stdout: debug leveldb = 0/0 2024-01-19T15:53:59.514 INFO:teuthology.orchestra.run.smithi181.stdout: debug lockdep = 0/0 2024-01-19T15:53:59.514 INFO:teuthology.orchestra.run.smithi181.stdout: debug mds = 0/0 2024-01-19T15:53:59.514 INFO:teuthology.orchestra.run.smithi181.stdout: debug mds balancer = 0/0 2024-01-19T15:53:59.514 INFO:teuthology.orchestra.run.smithi181.stdout: debug mds locker = 0/0 2024-01-19T15:53:59.514 INFO:teuthology.orchestra.run.smithi181.stdout: debug mds log = 0/0 2024-01-19T15:53:59.514 INFO:teuthology.orchestra.run.smithi181.stdout: debug mds log expire = 0/0 2024-01-19T15:53:59.515 INFO:teuthology.orchestra.run.smithi181.stdout: debug mds migrator = 0/0 2024-01-19T15:53:59.515 INFO:teuthology.orchestra.run.smithi181.stdout: debug memdb = 0/0 2024-01-19T15:53:59.515 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 0/0 2024-01-19T15:53:59.515 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgrc = 0/0 2024-01-19T15:53:59.515 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 0/0 2024-01-19T15:53:59.515 INFO:teuthology.orchestra.run.smithi181.stdout: debug monc = 0/0 2024-01-19T15:53:59.515 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 0/0 2024-01-19T15:53:59.515 INFO:teuthology.orchestra.run.smithi181.stdout: debug objclass = 0/0 2024-01-19T15:53:59.515 INFO:teuthology.orchestra.run.smithi181.stdout: debug objectcacher = 0/0 2024-01-19T15:53:59.515 INFO:teuthology.orchestra.run.smithi181.stdout: debug objecter = 0/0 2024-01-19T15:53:59.515 INFO:teuthology.orchestra.run.smithi181.stdout: debug optracker = 0/0 2024-01-19T15:53:59.515 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 0/0 2024-01-19T15:53:59.515 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 0/0 2024-01-19T15:53:59.515 INFO:teuthology.orchestra.run.smithi181.stdout: debug perfcounter = 0/0 2024-01-19T15:53:59.515 INFO:teuthology.orchestra.run.smithi181.stdout: debug rados = 0/0 2024-01-19T15:53:59.516 INFO:teuthology.orchestra.run.smithi181.stdout: debug rbd = 0/0 2024-01-19T15:53:59.516 INFO:teuthology.orchestra.run.smithi181.stdout: debug rbd mirror = 0/0 2024-01-19T15:53:59.516 INFO:teuthology.orchestra.run.smithi181.stdout: debug rbd replay = 0/0 2024-01-19T15:53:59.516 INFO:teuthology.orchestra.run.smithi181.stdout: debug refs = 0/0 2024-01-19T15:53:59.516 INFO:teuthology.orchestra.run.smithi181.stdout: debug rgw = 0/0 2024-01-19T15:53:59.516 INFO:teuthology.orchestra.run.smithi181.stdout: debug rgw sync = 0/0 2024-01-19T15:53:59.516 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 0/0 2024-01-19T15:53:59.516 INFO:teuthology.orchestra.run.smithi181.stdout: debug striper = 0/0 2024-01-19T15:53:59.516 INFO:teuthology.orchestra.run.smithi181.stdout: debug throttle = 0/0 2024-01-19T15:53:59.516 INFO:teuthology.orchestra.run.smithi181.stdout: debug timer = 0/0 2024-01-19T15:53:59.516 INFO:teuthology.orchestra.run.smithi181.stdout: debug tp = 0/0 2024-01-19T15:53:59.516 INFO:teuthology.orchestra.run.smithi181.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-01-19T15:53:59.516 INFO:teuthology.orchestra.run.smithi181.stdout: ms client mode = crc 2024-01-19T15:53:59.516 INFO:teuthology.orchestra.run.smithi181.stdout: ms cluster mode = crc 2024-01-19T15:53:59.516 INFO:teuthology.orchestra.run.smithi181.stdout: ms mon client mode = crc 2024-01-19T15:53:59.517 INFO:teuthology.orchestra.run.smithi181.stdout: ms mon cluster mode = crc 2024-01-19T15:53:59.517 INFO:teuthology.orchestra.run.smithi181.stdout: ms mon service mode = crc 2024-01-19T15:53:59.517 INFO:teuthology.orchestra.run.smithi181.stdout: ms service mode = crc 2024-01-19T15:53:59.517 INFO:teuthology.orchestra.run.smithi181.stdout: fsid = ced042d8-398d-4631-a7e9-4f6702eeddab 2024-01-19T15:53:59.517 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.517 INFO:teuthology.orchestra.run.smithi181.stdout:[osd] 2024-01-19T15:53:59.517 INFO:teuthology.orchestra.run.smithi181.stdout: osd journal size = 100 2024-01-19T15:53:59.517 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.517 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub load threshold = 5.0 2024-01-19T15:53:59.517 INFO:teuthology.orchestra.run.smithi181.stdout: osd scrub max interval = 600 2024-01-19T15:53:59.517 INFO:teuthology.orchestra.run.smithi181.stdout: osd mclock profile = high_recovery_ops 2024-01-19T15:53:59.517 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.517 INFO:teuthology.orchestra.run.smithi181.stdout: osd recover clone overlap = true 2024-01-19T15:53:59.517 INFO:teuthology.orchestra.run.smithi181.stdout: osd recovery max chunk = 1048576 2024-01-19T15:53:59.517 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.518 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug shutdown = true 2024-01-19T15:53:59.518 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug op order = true 2024-01-19T15:53:59.518 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug verify stray on activate = true 2024-01-19T15:53:59.518 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug trim objects = true 2024-01-19T15:53:59.518 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.518 INFO:teuthology.orchestra.run.smithi181.stdout: osd open classes on start = true 2024-01-19T15:53:59.518 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug pg log writeout = true 2024-01-19T15:53:59.518 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.518 INFO:teuthology.orchestra.run.smithi181.stdout: osd deep scrub update digest min age = 30 2024-01-19T15:53:59.518 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.518 INFO:teuthology.orchestra.run.smithi181.stdout: osd map max advance = 10 2024-01-19T15:53:59.518 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.518 INFO:teuthology.orchestra.run.smithi181.stdout: journal zero on create = true 2024-01-19T15:53:59.518 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.518 INFO:teuthology.orchestra.run.smithi181.stdout: filestore ondisk finisher threads = 3 2024-01-19T15:53:59.519 INFO:teuthology.orchestra.run.smithi181.stdout: filestore apply finisher threads = 3 2024-01-19T15:53:59.519 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.519 INFO:teuthology.orchestra.run.smithi181.stdout: bdev debug aio = true 2024-01-19T15:53:59.519 INFO:teuthology.orchestra.run.smithi181.stdout: osd debug misdirected ops = true 2024-01-19T15:53:59.519 INFO:teuthology.orchestra.run.smithi181.stdout: debug monc = 20 2024-01-19T15:53:59.519 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore block size = 96636764160 2024-01-19T15:53:59.519 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore compression algorithm = snappy 2024-01-19T15:53:59.519 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore compression mode = aggressive 2024-01-19T15:53:59.519 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore fsck on mount = True 2024-01-19T15:53:59.519 INFO:teuthology.orchestra.run.smithi181.stdout: bluestore rocksdb cf = False 2024-01-19T15:53:59.519 INFO:teuthology.orchestra.run.smithi181.stdout: crimson osd obc lru size = 10 2024-01-19T15:53:59.519 INFO:teuthology.orchestra.run.smithi181.stdout: crimson seastar smp = 3 2024-01-19T15:53:59.519 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluefs = 20 2024-01-19T15:53:59.519 INFO:teuthology.orchestra.run.smithi181.stdout: debug bluestore = 20 2024-01-19T15:53:59.519 INFO:teuthology.orchestra.run.smithi181.stdout: debug filestore = 0/0 2024-01-19T15:53:59.520 INFO:teuthology.orchestra.run.smithi181.stdout: debug journal = 0/0 2024-01-19T15:53:59.520 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 0/0 2024-01-19T15:53:59.520 INFO:teuthology.orchestra.run.smithi181.stdout: debug osd = 0/0 2024-01-19T15:53:59.520 INFO:teuthology.orchestra.run.smithi181.stdout: debug rocksdb = 10 2024-01-19T15:53:59.520 INFO:teuthology.orchestra.run.smithi181.stdout: err to stderr = True 2024-01-19T15:53:59.520 INFO:teuthology.orchestra.run.smithi181.stdout: log flush on exit = True 2024-01-19T15:53:59.520 INFO:teuthology.orchestra.run.smithi181.stdout: log to file = False 2024-01-19T15:53:59.520 INFO:teuthology.orchestra.run.smithi181.stdout: log to stderr = True 2024-01-19T15:53:59.520 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd backfillfull_ratio = 0.85 2024-01-19T15:53:59.520 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd full ratio = 0.9 2024-01-19T15:53:59.520 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd nearfull ratio = 0.8 2024-01-19T15:53:59.520 INFO:teuthology.orchestra.run.smithi181.stdout: osd failsafe full ratio = 0.95 2024-01-19T15:53:59.520 INFO:teuthology.orchestra.run.smithi181.stdout: osd objectstore = bluestore 2024-01-19T15:53:59.520 INFO:teuthology.orchestra.run.smithi181.stdout: osd shutdown pgref assert = True 2024-01-19T15:53:59.521 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.521 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr] 2024-01-19T15:53:59.521 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 1 2024-01-19T15:53:59.521 INFO:teuthology.orchestra.run.smithi181.stdout: debug mgr = 20 2024-01-19T15:53:59.521 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 20 2024-01-19T15:53:59.521 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-01-19T15:53:59.521 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-01-19T15:53:59.521 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-01-19T15:53:59.521 INFO:teuthology.orchestra.run.smithi181.stdout: mgr/telemetry/nag = false 2024-01-19T15:53:59.521 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.521 INFO:teuthology.orchestra.run.smithi181.stdout:[mon] 2024-01-19T15:53:59.521 INFO:teuthology.orchestra.run.smithi181.stdout: debug ms = 0/0 2024-01-19T15:53:59.521 INFO:teuthology.orchestra.run.smithi181.stdout: debug mon = 0/0 2024-01-19T15:53:59.521 INFO:teuthology.orchestra.run.smithi181.stdout: debug paxos = 0/0 2024-01-19T15:53:59.521 INFO:teuthology.orchestra.run.smithi181.stdout: debug auth = 20 2024-01-19T15:53:59.522 INFO:teuthology.orchestra.run.smithi181.stdout: mon data avail warn = 5 2024-01-19T15:53:59.522 INFO:teuthology.orchestra.run.smithi181.stdout: mon mgr mkfs grace = 240 2024-01-19T15:53:59.522 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min pgs per osd = 4 2024-01-19T15:53:59.522 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd reporter subtree level = osd 2024-01-19T15:53:59.522 INFO:teuthology.orchestra.run.smithi181.stdout: mon osd prime pg temp = true 2024-01-19T15:53:59.522 INFO:teuthology.orchestra.run.smithi181.stdout: mon reweight min bytes per osd = 10 2024-01-19T15:53:59.522 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.522 INFO:teuthology.orchestra.run.smithi181.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-19T15:53:59.522 INFO:teuthology.orchestra.run.smithi181.stdout: auth mon ticket ttl = 660 # 11m 2024-01-19T15:53:59.522 INFO:teuthology.orchestra.run.smithi181.stdout: auth service ticket ttl = 240 # 4m 2024-01-19T15:53:59.522 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.522 INFO:teuthology.orchestra.run.smithi181.stdout: # don't complain about insecure global_id in the test suite 2024-01-19T15:53:59.522 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-19T15:53:59.522 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-19T15:53:59.522 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.523 INFO:teuthology.orchestra.run.smithi181.stdout: # 1m isn't quite enough 2024-01-19T15:53:59.523 INFO:teuthology.orchestra.run.smithi181.stdout: mon_down_mkfs_grace = 2m 2024-01-19T15:53:59.523 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.523 INFO:teuthology.orchestra.run.smithi181.stdout: mon_warn_on_filestore_osds = false 2024-01-19T15:53:59.523 INFO:teuthology.orchestra.run.smithi181.stdout: osd pool default crimson = True 2024-01-19T15:53:59.523 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:53:59.523 INFO:teuthology.orchestra.run.smithi181.stdout:[client] 2024-01-19T15:53:59.523 INFO:teuthology.orchestra.run.smithi181.stdout: rgw cache enabled = true 2024-01-19T15:53:59.523 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable ops log = true 2024-01-19T15:53:59.523 INFO:teuthology.orchestra.run.smithi181.stdout: rgw enable usage log = true 2024-01-19T15:53:59.523 INFO:teuthology.orchestra.run.smithi181.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-19T15:53:59.523 INFO:teuthology.orchestra.run.smithi181.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-19T15:53:59.523 INFO:teuthology.orchestra.run.smithi181.stdout:[mon.a] 2024-01-19T15:53:59.524 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-19T15:53:59.524 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-19T15:53:59.585 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-19T15:53:59.585 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-01-19T15:53:59.585 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-19T15:53:59.600 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-01-19T15:53:59.600 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-19T15:53:59.654 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:53:59.655 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-01-19T15:53:59.655 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-19T15:53:59.655 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-19T15:53:59.738 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-01-19T15:53:59.738 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-19T15:53:59.793 INFO:tasks.ceph:Setting up mon nodes... 2024-01-19T15:53:59.793 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-19T15:53:59.793 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-19T15:53:59.893 INFO:teuthology.orchestra.run.smithi181.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-19T15:53:59.896 INFO:tasks.ceph:Setting up mds nodes... 2024-01-19T15:53:59.896 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-19T15:53:59.896 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-19T15:53:59.944 INFO:teuthology.orchestra.run.smithi181.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-19T15:53:59.960 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-19T15:53:59.960 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi181.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-01-19T15:53:59.960 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-19T15:53:59.986 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-19T15:53:59.986 INFO:tasks.ceph:role: osd.0 2024-01-19T15:53:59.986 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:53:59.987 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-19T15:54:00.060 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-19T15:54:00.061 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-19T15:54:00.061 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-19T15:54:00.061 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-19T15:54:00.061 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-19T15:54:00.061 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-01-19T15:54:00.061 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-19T15:54:00.061 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-19T15:54:00.061 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-19T15:54:00.061 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-19T15:54:00.585 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-01-19T15:54:00.587 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-01-19T15:54:00.588 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-19T15:54:00.706 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-19T15:54:00.739 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-19T15:54:00.807 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-19T15:54:00.807 INFO:tasks.ceph:role: osd.1 2024-01-19T15:54:00.807 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:54:00.807 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-19T15:54:00.881 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-19T15:54:00.881 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-19T15:54:00.881 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-19T15:54:00.881 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-19T15:54:00.881 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-19T15:54:00.881 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-01-19T15:54:00.881 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-19T15:54:00.881 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-19T15:54:00.882 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-19T15:54:00.882 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-19T15:54:01.331 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-01-19T15:54:01.333 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-01-19T15:54:01.333 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-19T15:54:01.415 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-19T15:54:01.486 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-19T15:54:01.552 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-01-19T15:54:01.553 INFO:tasks.ceph:role: osd.2 2024-01-19T15:54:01.553 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:54:01.553 DEBUG:teuthology.orchestra.run.smithi181:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-19T15:54:01.627 INFO:teuthology.orchestra.run.smithi181.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-19T15:54:01.627 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-19T15:54:01.627 INFO:teuthology.orchestra.run.smithi181.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-19T15:54:01.627 INFO:teuthology.orchestra.run.smithi181.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-19T15:54:01.627 INFO:teuthology.orchestra.run.smithi181.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-19T15:54:01.627 INFO:teuthology.orchestra.run.smithi181.stdout: = sunit=0 swidth=0 blks 2024-01-19T15:54:01.627 INFO:teuthology.orchestra.run.smithi181.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-19T15:54:01.627 INFO:teuthology.orchestra.run.smithi181.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-19T15:54:01.628 INFO:teuthology.orchestra.run.smithi181.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-19T15:54:01.628 INFO:teuthology.orchestra.run.smithi181.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-19T15:54:02.147 INFO:teuthology.orchestra.run.smithi181.stdout:Discarding blocks...Done. 2024-01-19T15:54:02.149 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi181.front.sepia.ceph.com -o noatime 2024-01-19T15:54:02.149 DEBUG:teuthology.orchestra.run.smithi181:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-19T15:54:02.287 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-19T15:54:02.319 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-19T15:54:02.897 INFO:teuthology.orchestra.run.smithi181.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-19T15:54:02.898 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:02,907 seastar - Reactor backend: linux-aio 2024-01-19T15:54:02.915 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:02,925 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-19T15:54:02.915 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:02,925 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-19T15:54:02.916 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:02,925 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-19T15:54:02.916 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:02,925 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-19T15:54:02.917 INFO:teuthology.orchestra.run.smithi181.stderr:==27807==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-19T15:54:02.924 INFO:teuthology.orchestra.run.smithi181.stderr:==27807==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffcbfc83000; bottom 0x7effd6bcd000; size: 0x00fce90b6000 (1086241595392) 2024-01-19T15:54:02.924 INFO:teuthology.orchestra.run.smithi181.stderr:False positive error reports may follow 2024-01-19T15:54:02.924 INFO:teuthology.orchestra.run.smithi181.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-19T15:54:03.458 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-01-19T15:54:03.458 INFO:teuthology.orchestra.run.smithi181.stderr:Suppressions used: 2024-01-19T15:54:03.458 INFO:teuthology.orchestra.run.smithi181.stderr: count bytes template 2024-01-19T15:54:03.458 INFO:teuthology.orchestra.run.smithi181.stderr: 1 8 MallocExtension::Register 2024-01-19T15:54:03.458 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-01-19T15:54:03.458 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-01-19T15:54:03.516 INFO:teuthology.orchestra.run.smithi181.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-19T15:54:03.516 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:03,526 seastar - Reactor backend: linux-aio 2024-01-19T15:54:03.534 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:03,544 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-19T15:54:03.534 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:03,544 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-19T15:54:03.534 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:03,544 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-19T15:54:03.541 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:03,551 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-19T15:54:03.544 INFO:teuthology.orchestra.run.smithi181.stderr:==27805==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-19T15:54:03.550 INFO:teuthology.orchestra.run.smithi181.stderr:==27805==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffcbfc83000; bottom 0x7effd49b6000; size: 0x00fceb2cd000 (1086277341184) 2024-01-19T15:54:03.550 INFO:teuthology.orchestra.run.smithi181.stderr:False positive error reports may follow 2024-01-19T15:54:03.550 INFO:teuthology.orchestra.run.smithi181.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-19T15:54:03.635 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2e81660e 0x2e8179bc 0x2e818eee 0x2e819272 0x2e8193bc 0x2e819812 0x54daf 0xce1ca 0x588c4 0xdb9f8 0x278181d6 0x2781d3fe 0x2781fc63 0x2763945d 0x1f9992f4 0x1f7111b4 0x1f83c2a9 0x2ebc1255 2024-01-19T15:54:03.635 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa5c79b17 0xffffffffa5c75447 0xffffffffa5e000ea 2024-01-19T15:54:07.567 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-01-19T15:54:07.568 INFO:teuthology.orchestra.run.smithi181.stderr:Suppressions used: 2024-01-19T15:54:07.568 INFO:teuthology.orchestra.run.smithi181.stderr: count bytes template 2024-01-19T15:54:07.568 INFO:teuthology.orchestra.run.smithi181.stderr: 1 8 MallocExtension::Register 2024-01-19T15:54:07.568 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-01-19T15:54:07.568 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-01-19T15:54:07.568 INFO:teuthology.orchestra.run.smithi181.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid ced042d8-398d-4631-a7e9-4f6702eeddab 2024-01-19T15:54:07.605 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-19T15:54:07.632 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-19T15:54:08.156 INFO:teuthology.orchestra.run.smithi181.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-19T15:54:08.157 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:08,166 seastar - Reactor backend: linux-aio 2024-01-19T15:54:08.176 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:08,186 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-19T15:54:08.176 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:08,186 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-19T15:54:08.176 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:08,186 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-19T15:54:08.177 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:08,186 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-19T15:54:08.178 INFO:teuthology.orchestra.run.smithi181.stderr:==27989==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-19T15:54:08.184 INFO:teuthology.orchestra.run.smithi181.stderr:==27989==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fffcf005000; bottom 0x7f37cd8e6000; size: 0x00c80171f000 (859017703424) 2024-01-19T15:54:08.185 INFO:teuthology.orchestra.run.smithi181.stderr:False positive error reports may follow 2024-01-19T15:54:08.185 INFO:teuthology.orchestra.run.smithi181.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-19T15:54:08.692 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-01-19T15:54:08.692 INFO:teuthology.orchestra.run.smithi181.stderr:Suppressions used: 2024-01-19T15:54:08.692 INFO:teuthology.orchestra.run.smithi181.stderr: count bytes template 2024-01-19T15:54:08.692 INFO:teuthology.orchestra.run.smithi181.stderr: 1 8 MallocExtension::Register 2024-01-19T15:54:08.692 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-01-19T15:54:08.693 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-01-19T15:54:08.746 INFO:teuthology.orchestra.run.smithi181.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-19T15:54:08.746 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:08,756 seastar - Reactor backend: linux-aio 2024-01-19T15:54:08.780 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:08,790 [shard 2] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-19T15:54:08.781 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:08,790 [shard 2] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-19T15:54:08.781 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:08,790 [shard 2] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-19T15:54:08.781 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:08,790 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-19T15:54:08.783 INFO:teuthology.orchestra.run.smithi181.stderr:==27987==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-19T15:54:08.789 INFO:teuthology.orchestra.run.smithi181.stderr:==27987==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7fffcf005000; bottom 0x7f37cb6cf000; size: 0x00c803936000 (859053449216) 2024-01-19T15:54:08.789 INFO:teuthology.orchestra.run.smithi181.stderr:False positive error reports may follow 2024-01-19T15:54:08.789 INFO:teuthology.orchestra.run.smithi181.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-19T15:54:12.787 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-01-19T15:54:12.788 INFO:teuthology.orchestra.run.smithi181.stderr:Suppressions used: 2024-01-19T15:54:12.788 INFO:teuthology.orchestra.run.smithi181.stderr: count bytes template 2024-01-19T15:54:12.788 INFO:teuthology.orchestra.run.smithi181.stderr: 1 8 MallocExtension::Register 2024-01-19T15:54:12.788 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-01-19T15:54:12.788 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-01-19T15:54:12.788 INFO:teuthology.orchestra.run.smithi181.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid ced042d8-398d-4631-a7e9-4f6702eeddab 2024-01-19T15:54:12.823 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-19T15:54:12.850 DEBUG:teuthology.orchestra.run.smithi181:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-19T15:54:13.373 INFO:teuthology.orchestra.run.smithi181.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-19T15:54:13.374 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:13,383 seastar - Reactor backend: linux-aio 2024-01-19T15:54:13.389 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:13,399 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-19T15:54:13.390 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:13,399 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-19T15:54:13.390 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:13,399 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-19T15:54:13.390 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:13,399 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-19T15:54:13.392 INFO:teuthology.orchestra.run.smithi181.stderr:==28171==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-19T15:54:13.398 INFO:teuthology.orchestra.run.smithi181.stderr:==28171==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffea27f7000; bottom 0x7fe195a3c000; size: 0x001d0cdbb000 (124769775616) 2024-01-19T15:54:13.398 INFO:teuthology.orchestra.run.smithi181.stderr:False positive error reports may follow 2024-01-19T15:54:13.398 INFO:teuthology.orchestra.run.smithi181.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-19T15:54:13.424 INFO:teuthology.orchestra.run.smithi181.stderr:Reactor stalled for 32 ms on shard 0. Backtrace: 0x45d5d 0x2e81660e 0x2e8179bc 0x2e818eee 0x2e819272 0x2e8193bc 0x2e819812 0x54daf 0x240eeb3a 0x240f3e6f 0x240f4102 0x241ae127 0x23f6f8f2 0x23ff1499 0x25eb7aae 0x25ebf5e2 0x25ec1f2f 0x25ec30f4 0x23ace913 0x23aeb87a 0x23b60925 0x2e788fc7 0x2e7ec1f8 0x2ea921fd 0x2ea94047 0x2e4454a4 0x2e4473ad 0x1f937895 0x1f93cf5b 0x1f7151d2 0x3feaf 0x3ff5f 0x1f2d5b64 2024-01-19T15:54:13.424 INFO:teuthology.orchestra.run.smithi181.stderr:kernel callstack: 2024-01-19T15:54:13.916 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-01-19T15:54:13.917 INFO:teuthology.orchestra.run.smithi181.stderr:Suppressions used: 2024-01-19T15:54:13.917 INFO:teuthology.orchestra.run.smithi181.stderr: count bytes template 2024-01-19T15:54:13.917 INFO:teuthology.orchestra.run.smithi181.stderr: 1 8 MallocExtension::Register 2024-01-19T15:54:13.917 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-01-19T15:54:13.917 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-01-19T15:54:13.973 INFO:teuthology.orchestra.run.smithi181.stderr:WARNING: debug mode. Not for benchmarking or production 2024-01-19T15:54:13.973 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:13,983 seastar - Reactor backend: linux-aio 2024-01-19T15:54:14.001 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:14,010 [shard 1] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000 2024-01-19T15:54:14.001 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:14,010 [shard 1] seastar - IO queue uses 0.75ms latency goal for device 0 2024-01-19T15:54:14.001 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:14,010 [shard 1] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647 2024-01-19T15:54:14.008 INFO:teuthology.orchestra.run.smithi181.stderr:INFO 2024-01-19 15:54:14,018 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000 2024-01-19T15:54:14.011 INFO:teuthology.orchestra.run.smithi181.stderr:==28169==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-01-19T15:54:14.016 INFO:teuthology.orchestra.run.smithi181.stderr:==28169==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0x7ffea27f7000; bottom 0x7fe193825000; size: 0x001d0efd2000 (124805521408) 2024-01-19T15:54:14.017 INFO:teuthology.orchestra.run.smithi181.stderr:False positive error reports may follow 2024-01-19T15:54:14.017 INFO:teuthology.orchestra.run.smithi181.stderr:For details see https://github.com/google/sanitizers/issues/189 2024-01-19T15:54:18.024 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-01-19T15:54:18.025 INFO:teuthology.orchestra.run.smithi181.stderr:Suppressions used: 2024-01-19T15:54:18.025 INFO:teuthology.orchestra.run.smithi181.stderr: count bytes template 2024-01-19T15:54:18.025 INFO:teuthology.orchestra.run.smithi181.stderr: 1 8 MallocExtension::Register 2024-01-19T15:54:18.025 INFO:teuthology.orchestra.run.smithi181.stderr:----------------------------------------------------- 2024-01-19T15:54:18.025 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-01-19T15:54:18.025 INFO:teuthology.orchestra.run.smithi181.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid ced042d8-398d-4631-a7e9-4f6702eeddab 2024-01-19T15:54:18.063 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-19T15:54:18.089 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-19T15:54:18.090 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-01-19T15:54:18.090 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-19T15:54:18.157 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-01-19T15:54:18.157 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-19T15:54:18.224 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-01-19T15:54:18.224 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-19T15:54:18.291 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-01-19T15:54:18.291 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-19T15:54:18.358 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-01-19T15:54:18.358 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-19T15:54:18.413 INFO:tasks.ceph:Adding keys to all mons... 2024-01-19T15:54:18.413 DEBUG:teuthology.orchestra.run.smithi181:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-19T15:54:18.478 INFO:teuthology.orchestra.run.smithi181.stdout:[mgr.x] 2024-01-19T15:54:18.478 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQAXm6pl9zDVNRAAat2HcME7oQPtaEU01XK4LA== 2024-01-19T15:54:18.478 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.0] 2024-01-19T15:54:18.479 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQAbm6pl8489JBAAKsEgydU+PY+wa1RsTPK6Mw== 2024-01-19T15:54:18.479 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.1] 2024-01-19T15:54:18.479 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQAgm6plNRNfMhAAigA5SdWHcuxeP1avP7LDPQ== 2024-01-19T15:54:18.479 INFO:teuthology.orchestra.run.smithi181.stdout:[osd.2] 2024-01-19T15:54:18.479 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQAmm6plhYRXBBAAkdS8MCCoAfFXP4yUQu2jxw== 2024-01-19T15:54:18.479 INFO:teuthology.orchestra.run.smithi181.stdout:[client.0] 2024-01-19T15:54:18.479 INFO:teuthology.orchestra.run.smithi181.stdout: key = AQAXm6plWePcOBAAUDxBqDOtkCaIQjfo04hjmw== 2024-01-19T15:54:18.480 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-19T15:54:18.578 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-19T15:54:18.635 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-19T15:54:18.694 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-19T15:54:18.752 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-19T15:54:18.812 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-19T15:54:18.812 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-19T15:54:18.839 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-19T15:54:18.939 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:18.948+0000 7f155e23a9c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:18.939 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:18.948+0000 7f155e23a9c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:18.951 DEBUG:teuthology.orchestra.run.smithi181:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-19T15:54:18.978 DEBUG:teuthology.orchestra.run.smithi181:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-19T15:54:19.032 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-19T15:54:19.033 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-19T15:54:19.033 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-01-19T15:54:19.075 INFO:tasks.ceph.mon.a:Started 2024-01-19T15:54:19.075 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-19T15:54:19.075 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-19T15:54:19.075 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-01-19T15:54:19.077 INFO:tasks.ceph.mgr.x:Started 2024-01-19T15:54:19.078 INFO:tasks.ceph:Setting crush tunables to default 2024-01-19T15:54:19.078 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd crush tunables default 2024-01-19T15:54:19.156 INFO:tasks.ceph.mon.a.smithi181.stderr:2024-01-19T15:54:19.165+0000 7f4e226149c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:19.157 INFO:tasks.ceph.mon.a.smithi181.stderr:2024-01-19T15:54:19.165+0000 7f4e226149c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:19.157 INFO:tasks.ceph.mon.a.smithi181.stderr:2024-01-19T15:54:19.165+0000 7f4e226149c0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:19.158 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:19.167+0000 7f0da7686f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:19.186 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:19.190+0000 7f7550d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:19.215 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:19.224+0000 7f7550d9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:19.284 INFO:teuthology.orchestra.run.smithi181.stderr:adjusted tunables profile to default 2024-01-19T15:54:19.295 INFO:tasks.ceph:check_enable_crimson: True 2024-01-19T15:54:19.295 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-01-19T15:54:19.296 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-01-19T15:54:19.362 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:19.370+0000 7f0da7686f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:19.362 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:19.371+0000 7f0da7686f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:19.400 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:19.405+0000 7f063def7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:19.402 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:19.411+0000 7f063def7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:19.483 INFO:tasks.ceph.mon.a.smithi181.stderr:2024-01-19T15:54:19.491+0000 7f4e1a5f1640 -1 WARNING: experimental feature 'crimson' is enabled 2024-01-19T15:54:19.483 INFO:tasks.ceph.mon.a.smithi181.stderr:Please be aware that this feature is experimental, untested, 2024-01-19T15:54:19.483 INFO:tasks.ceph.mon.a.smithi181.stderr:unsupported, and may result in data corruption, data loss, 2024-01-19T15:54:19.483 INFO:tasks.ceph.mon.a.smithi181.stderr:and/or irreparable damage to your cluster. Do not use 2024-01-19T15:54:19.483 INFO:tasks.ceph.mon.a.smithi181.stderr:feature with important data. 2024-01-19T15:54:19.483 INFO:tasks.ceph.mon.a.smithi181.stderr: 2024-01-19T15:54:19.836 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:19.844+0000 7f0da7686f00 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-19T15:54:19.887 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:19.896+0000 7f0da7686f00 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-19T15:54:20.297 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-19T15:54:20.297 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-01-19T15:54:20.297 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-19T15:54:20.324 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-01-19T15:54:20.324 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-19T15:54:20.391 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-01-19T15:54:20.392 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-19T15:54:20.411 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:20.420+0000 7f0da7686f00 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-19T15:54:20.437 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new dea89925-9629-44d2-a93e-2ae62d0bf8dc 0 2024-01-19T15:54:20.466 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:20.475+0000 7f0da7686f00 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-19T15:54:20.513 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:20.522+0000 7f0da7686f00 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-19T15:54:20.571 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:20.579+0000 7f0da7686f00 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-19T15:54:20.572 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:20.576+0000 7fdc7f0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:20.574 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:20.583+0000 7fdc7f0ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:20.615 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:20.624+0000 7f0da7686f00 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-19T15:54:20.650 INFO:teuthology.orchestra.run.smithi181.stdout:0 2024-01-19T15:54:20.662 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 631f59bd-fbd2-4091-a917-aef1ec0320d5 1 2024-01-19T15:54:20.722 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:20.731+0000 7f0da7686f00 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-19T15:54:20.768 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:20.771+0000 7f21d3b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:20.770 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:20.778+0000 7f21d3b5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:20.796 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:20.805+0000 7f0da7686f00 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-19T15:54:20.843 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:20.851+0000 7f0da7686f00 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-19T15:54:20.846 INFO:teuthology.orchestra.run.smithi181.stdout:1 2024-01-19T15:54:20.857 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd new 06f7135b-1b19-4419-878a-e92b38573dc8 2 2024-01-19T15:54:20.899 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:20.908+0000 7f0da7686f00 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-19T15:54:20.949 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:20.958+0000 7f0da7686f00 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-19T15:54:20.963 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:20.967+0000 7fee92d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:20.965 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:20.974+0000 7fee92d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:21.041 INFO:teuthology.orchestra.run.smithi181.stdout:2 2024-01-19T15:54:21.053 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-01-19T15:54:21.125 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-19T15:54:21.125 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-01-19T15:54:21.167 INFO:tasks.ceph.osd.0:Started 2024-01-19T15:54:21.168 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-01-19T15:54:21.199 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-19T15:54:21.199 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-01-19T15:54:21.241 INFO:tasks.ceph.osd.1:Started 2024-01-19T15:54:21.242 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-01-19T15:54:21.276 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-19T15:54:21.276 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-01-19T15:54:21.318 INFO:tasks.ceph.osd.2:Started 2024-01-19T15:54:21.319 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-19T15:54:21.376 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:21.385+0000 7f0da7686f00 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-19T15:54:21.520 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:21.522+0000 7f52f535e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:21.524 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:21.533+0000 7f52f535e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:21.592 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:21.601+0000 7f0da7686f00 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-19T15:54:21.663 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:54:21.663 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":6,"fsid":"ced042d8-398d-4631-a7e9-4f6702eeddab","created":"2024-01-19T15:54:19.216523+0000","modified":"2024-01-19T15:54:21.050997+0000","last_up_change":"0.000000","last_in_change":"2024-01-19T15:54:21.050997+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"dea89925-9629-44d2-a93e-2ae62d0bf8dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"631f59bd-fbd2-4091-a917-aef1ec0320d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"06f7135b-1b19-4419-878a-e92b38573dc8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-19T15:54:21.687 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-19T15:54:21.687 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-19T15:54:21.735 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:21.744+0000 7f0da7686f00 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-19T15:54:21.787 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:21.796+0000 7f0da7686f00 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-19T15:54:21.893 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:21.902+0000 7f0da7686f00 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-19T15:54:21.989 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-19T15:54:21.996 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:22.005+0000 7f0da7686f00 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-19T15:54:22.088 INFO:teuthology.misc.health.smithi181.stderr:2024-01-19T15:54:22.094+0000 7f23fcb1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:22.091 INFO:teuthology.misc.health.smithi181.stderr:2024-01-19T15:54:22.100+0000 7f23fcb1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:22.151 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:22.159+0000 7f0da7686f00 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-19T15:54:22.157 INFO:teuthology.misc.health.smithi181.stdout: 2024-01-19T15:54:22.157 INFO:teuthology.misc.health.smithi181.stdout:{"epoch":6,"fsid":"ced042d8-398d-4631-a7e9-4f6702eeddab","created":"2024-01-19T15:54:19.216523+0000","modified":"2024-01-19T15:54:21.050997+0000","last_up_change":"0.000000","last_in_change":"2024-01-19T15:54:21.050997+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"dea89925-9629-44d2-a93e-2ae62d0bf8dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"631f59bd-fbd2-4091-a917-aef1ec0320d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"06f7135b-1b19-4419-878a-e92b38573dc8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-19T15:54:22.167 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-01-19T15:54:22.198 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:22.207+0000 7f0da7686f00 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-19T15:54:22.352 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:22.360+0000 7f0da7686f00 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-19T15:54:23.213 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:23.222+0000 7f0da7686f00 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-19T15:54:23.516 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:23.525+0000 7f0da7686f00 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-19T15:54:23.570 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:23.579+0000 7f0da7686f00 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-19T15:54:23.675 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:23.684+0000 7f0da7686f00 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-19T15:54:24.695 INFO:tasks.ceph.mgr.x.smithi181.stderr:2024-01-19T15:54:24.703+0000 7f0d47c6e640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-01-19T15:54:28.469 DEBUG:teuthology.orchestra.run.smithi181:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-19T15:54:28.575 INFO:teuthology.misc.health.smithi181.stderr:2024-01-19T15:54:28.581+0000 7f381591e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:28.577 INFO:teuthology.misc.health.smithi181.stderr:2024-01-19T15:54:28.586+0000 7f381591e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:28.760 INFO:teuthology.misc.health.smithi181.stdout: 2024-01-19T15:54:28.760 INFO:teuthology.misc.health.smithi181.stdout:{"epoch":9,"fsid":"ced042d8-398d-4631-a7e9-4f6702eeddab","created":"2024-01-19T15:54:19.216523+0000","modified":"2024-01-19T15:54:28.710069+0000","last_up_change":"2024-01-19T15:54:26.705109+0000","last_in_change":"2024-01-19T15:54:21.050997+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-19T15:54:27.741078+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"9","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dea89925-9629-44d2-a93e-2ae62d0bf8dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":1840670903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1840670903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1840670903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":1840670903}]},"public_addr":"172.21.15.181:6803/1840670903","cluster_addr":"172.21.15.181:6802/1840670903","heartbeat_back_addr":"172.21.15.181:6808/1840670903","heartbeat_front_addr":"172.21.15.181:6809/1840670903","state":["exists","up"]},{"osd":1,"uuid":"631f59bd-fbd2-4091-a917-aef1ec0320d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":83691468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6805","nonce":83691468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6811","nonce":83691468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":83691468}]},"public_addr":"172.21.15.181:6804/83691468","cluster_addr":"172.21.15.181:6805/83691468","heartbeat_back_addr":"172.21.15.181:6811/83691468","heartbeat_front_addr":"172.21.15.181:6810/83691468","state":["exists","up"]},{"osd":2,"uuid":"06f7135b-1b19-4419-878a-e92b38573dc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6807","nonce":2796840369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2796840369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":2796840369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2796840369}]},"public_addr":"172.21.15.181:6807/2796840369","cluster_addr":"172.21.15.181:6806/2796840369","heartbeat_back_addr":"172.21.15.181:6813/2796840369","heartbeat_front_addr":"172.21.15.181:6812/2796840369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-19T15:54:28.771 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-01-19T15:54:28.771 INFO:tasks.ceph:Creating RBD pool 2024-01-19T15:54:28.771 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd pool create rbd 8 2024-01-19T15:54:28.876 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:28.879+0000 7f6384b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:28.877 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:28.886+0000 7f6384b1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:29.701 INFO:teuthology.orchestra.run.smithi181.stderr:pool 'rbd' created 2024-01-19T15:54:29.719 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-01-19T15:54:29.841 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:29.841+0000 7f9cd539e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:29.843 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:29.852+0000 7f9cd539e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:30.713 INFO:teuthology.orchestra.run.smithi181.stderr:enabled application 'rbd' on pool 'rbd' 2024-01-19T15:54:30.726 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-19T15:54:30.726 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-19T15:54:30.726 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-19T15:54:30.727 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-19T15:54:30.727 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-19T15:54:30.846 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:30.850+0000 7fc83f79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:30.848 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:30.857+0000 7fc83f79e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:31.079 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:54:31.090 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":4,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":3834336927},{"type":"v1","addr":"172.21.15.181:6801","nonce":3834336927}]},"active_addr":"172.21.15.181:6801/3834336927","active_change":"2024-01-19T15:54:23.698527+0000","active_mgr_features":4540701272860131327,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":238276325}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":585436118}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.181:0","nonce":2432730330}]}]} 2024-01-19T15:54:31.094 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-19T15:54:31.094 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-19T15:54:31.094 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-19T15:54:31.200 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:31.206+0000 7f93078f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:31.202 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:31.211+0000 7f93078f7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:31.378 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:54:31.378 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":11,"fsid":"ced042d8-398d-4631-a7e9-4f6702eeddab","created":"2024-01-19T15:54:19.216523+0000","modified":"2024-01-19T15:54:30.720196+0000","last_up_change":"2024-01-19T15:54:26.705109+0000","last_in_change":"2024-01-19T15:54:21.050997+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-19T15:54:27.741078+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-19T15:54:29.079306+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dea89925-9629-44d2-a93e-2ae62d0bf8dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":1840670903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1840670903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1840670903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":1840670903}]},"public_addr":"172.21.15.181:6803/1840670903","cluster_addr":"172.21.15.181:6802/1840670903","heartbeat_back_addr":"172.21.15.181:6808/1840670903","heartbeat_front_addr":"172.21.15.181:6809/1840670903","state":["exists","up"]},{"osd":1,"uuid":"631f59bd-fbd2-4091-a917-aef1ec0320d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":83691468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6805","nonce":83691468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6811","nonce":83691468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":83691468}]},"public_addr":"172.21.15.181:6804/83691468","cluster_addr":"172.21.15.181:6805/83691468","heartbeat_back_addr":"172.21.15.181:6811/83691468","heartbeat_front_addr":"172.21.15.181:6810/83691468","state":["exists","up"]},{"osd":2,"uuid":"06f7135b-1b19-4419-878a-e92b38573dc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6807","nonce":2796840369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2796840369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":2796840369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2796840369}]},"public_addr":"172.21.15.181:6807/2796840369","cluster_addr":"172.21.15.181:6806/2796840369","heartbeat_back_addr":"172.21.15.181:6813/2796840369","heartbeat_front_addr":"172.21.15.181:6812/2796840369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-19T15:54:31.390 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-19T15:54:31.390 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-19T15:54:31.495 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:31.501+0000 7f689a82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:31.497 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:31.506+0000 7f689a82d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:31.673 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:54:31.673 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":11,"fsid":"ced042d8-398d-4631-a7e9-4f6702eeddab","created":"2024-01-19T15:54:19.216523+0000","modified":"2024-01-19T15:54:30.720196+0000","last_up_change":"2024-01-19T15:54:26.705109+0000","last_in_change":"2024-01-19T15:54:21.050997+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-19T15:54:27.741078+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-01-19T15:54:29.079306+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dea89925-9629-44d2-a93e-2ae62d0bf8dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6803","nonce":1840670903}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1840670903}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":1840670903}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6809","nonce":1840670903}]},"public_addr":"172.21.15.181:6803/1840670903","cluster_addr":"172.21.15.181:6802/1840670903","heartbeat_back_addr":"172.21.15.181:6808/1840670903","heartbeat_front_addr":"172.21.15.181:6809/1840670903","state":["exists","up"]},{"osd":1,"uuid":"631f59bd-fbd2-4091-a917-aef1ec0320d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":83691468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6805","nonce":83691468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6811","nonce":83691468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":83691468}]},"public_addr":"172.21.15.181:6804/83691468","cluster_addr":"172.21.15.181:6805/83691468","heartbeat_back_addr":"172.21.15.181:6811/83691468","heartbeat_front_addr":"172.21.15.181:6810/83691468","state":["exists","up"]},{"osd":2,"uuid":"06f7135b-1b19-4419-878a-e92b38573dc8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6807","nonce":2796840369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":2796840369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6813","nonce":2796840369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":2796840369}]},"public_addr":"172.21.15.181:6807/2796840369","cluster_addr":"172.21.15.181:6806/2796840369","heartbeat_back_addr":"172.21.15.181:6813/2796840369","heartbeat_front_addr":"172.21.15.181:6812/2796840369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-19T15:54:31.686 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-19T15:54:31.687 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-19T15:54:31.687 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-19T15:54:31.804 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:31.809+0000 7f03cbc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:31.806 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:31.811+0000 7f589ac17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:31.806 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:31.811+0000 7ffbb0f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:31.807 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:31.816+0000 7f03cbc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:31.808 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:31.817+0000 7f589ac17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:31.809 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:31.818+0000 7ffbb0f1e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:31.832 INFO:teuthology.orchestra.run.smithi181.stdout:34359738373 2024-01-19T15:54:31.832 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-19T15:54:31.834 INFO:teuthology.orchestra.run.smithi181.stdout:34359738373 2024-01-19T15:54:31.834 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-19T15:54:31.835 INFO:teuthology.orchestra.run.smithi181.stdout:34359738373 2024-01-19T15:54:31.836 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-19T15:54:31.941 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:31.946+0000 7fa255d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:31.943 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:31.952+0000 7fa255d5e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:31.987 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:31.992+0000 7f581422d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:31.989 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:31.994+0000 7f5847a17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:31.990 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:31.999+0000 7f581422d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:31.991 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:32.000+0000 7f5847a17640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:32.135 INFO:teuthology.orchestra.run.smithi181.stdout:34359738372 2024-01-19T15:54:32.147 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.2 2024-01-19T15:54:32.175 INFO:teuthology.orchestra.run.smithi181.stdout:34359738372 2024-01-19T15:54:32.185 INFO:teuthology.orchestra.run.smithi181.stdout:34359738372 2024-01-19T15:54:32.188 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2024-01-19T15:54:32.197 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.0 2024-01-19T15:54:33.148 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-19T15:54:33.189 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-19T15:54:33.198 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-19T15:54:33.270 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:33.273+0000 7f9d5631e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:33.272 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:33.281+0000 7f9d5631e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:33.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:33.313+0000 7f511e2de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:33.311 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:33.320+0000 7f511e2de640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:33.349 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:33.354+0000 7fed36006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:33.351 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:33.360+0000 7fed36006640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:33.466 INFO:teuthology.orchestra.run.smithi181.stdout:34359738372 2024-01-19T15:54:33.478 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.2 2024-01-19T15:54:33.500 INFO:teuthology.orchestra.run.smithi181.stdout:34359738372 2024-01-19T15:54:33.512 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2024-01-19T15:54:33.536 INFO:teuthology.orchestra.run.smithi181.stdout:34359738372 2024-01-19T15:54:33.549 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.0 2024-01-19T15:54:34.479 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-19T15:54:34.514 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-19T15:54:34.549 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-19T15:54:34.602 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:34.605+0000 7f26890ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:34.604 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:34.613+0000 7f26890ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:34.638 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:34.643+0000 7f2b83b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:34.640 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:34.649+0000 7f2b83b9e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:34.681 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:34.686+0000 7feedb86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:34.683 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:34.692+0000 7feedb86d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:34.816 INFO:teuthology.orchestra.run.smithi181.stdout:34359738373 2024-01-19T15:54:34.827 INFO:teuthology.orchestra.run.smithi181.stdout:34359738373 2024-01-19T15:54:34.829 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2024-01-19T15:54:34.830 DEBUG:teuthology.parallel:result is None 2024-01-19T15:54:34.840 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2024-01-19T15:54:34.840 DEBUG:teuthology.parallel:result is None 2024-01-19T15:54:34.889 INFO:teuthology.orchestra.run.smithi181.stdout:34359738373 2024-01-19T15:54:34.901 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2024-01-19T15:54:34.901 DEBUG:teuthology.parallel:result is None 2024-01-19T15:54:34.901 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-19T15:54:34.901 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-19T15:54:35.007 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:35.013+0000 7f0e9631e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:35.009 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:35.017+0000 7f0e9631e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:35.184 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:54:35.184 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-01-19T15:54:35.196 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-01-19T15:54:33.702455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80900,"kb_used_data":404,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034620,"statfs":{"total":289910292480,"available":289827450880,"internally_reserved":0,"allocated":413696,"data_stored":1248195,"data_compressed":59922,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.948327"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:54:31.769615+0000","last_change":"2024-01-19T15:54:30.825728+0000","last_active":"2024-01-19T15:54:31.769615+0000","last_peered":"2024-01-19T15:54:31.769615+0000","last_clean":"2024-01-19T15:54:31.769615+0000","last_became_active":"2024-01-19T15:54:30.823983+0000","last_became_peered":"2024-01-19T15:54:30.823983+0000","last_unstale":"2024-01-19T15:54:31.769615+0000","last_undegraded":"2024-01-19T15:54:31.769615+0000","last_fullsized":"2024-01-19T15:54:31.769615+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_clean_scrub_stamp":"2024-01-19T15:54:29.710724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:54:31.764839+0000","last_change":"2024-01-19T15:54:30.837459+0000","last_active":"2024-01-19T15:54:31.764839+0000","last_peered":"2024-01-19T15:54:31.764839+0000","last_clean":"2024-01-19T15:54:31.764839+0000","last_became_active":"2024-01-19T15:54:30.835538+0000","last_became_peered":"2024-01-19T15:54:30.835538+0000","last_unstale":"2024-01-19T15:54:31.764839+0000","last_undegraded":"2024-01-19T15:54:31.764839+0000","last_fullsized":"2024-01-19T15:54:31.764839+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_clean_scrub_stamp":"2024-01-19T15:54:29.710724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:54:31.770478+0000","last_change":"2024-01-19T15:54:30.798448+0000","last_active":"2024-01-19T15:54:31.770478+0000","last_peered":"2024-01-19T15:54:31.770478+0000","last_clean":"2024-01-19T15:54:31.770478+0000","last_became_active":"2024-01-19T15:54:30.791023+0000","last_became_peered":"2024-01-19T15:54:30.791023+0000","last_unstale":"2024-01-19T15:54:31.770478+0000","last_undegraded":"2024-01-19T15:54:31.770478+0000","last_fullsized":"2024-01-19T15:54:31.770478+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_clean_scrub_stamp":"2024-01-19T15:54:29.710724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:54:31.766726+0000","last_change":"2024-01-19T15:54:30.840527+0000","last_active":"2024-01-19T15:54:31.766726+0000","last_peered":"2024-01-19T15:54:31.766726+0000","last_clean":"2024-01-19T15:54:31.766726+0000","last_became_active":"2024-01-19T15:54:30.836950+0000","last_became_peered":"2024-01-19T15:54:30.836950+0000","last_unstale":"2024-01-19T15:54:31.766726+0000","last_undegraded":"2024-01-19T15:54:31.766726+0000","last_fullsized":"2024-01-19T15:54:31.766726+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_clean_scrub_stamp":"2024-01-19T15:54:29.710724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:54:31.769038+0000","last_change":"2024-01-19T15:54:30.818720+0000","last_active":"2024-01-19T15:54:31.769038+0000","last_peered":"2024-01-19T15:54:31.769038+0000","last_clean":"2024-01-19T15:54:31.769038+0000","last_became_active":"2024-01-19T15:54:30.816411+0000","last_became_peered":"2024-01-19T15:54:30.816411+0000","last_unstale":"2024-01-19T15:54:31.769038+0000","last_undegraded":"2024-01-19T15:54:31.769038+0000","last_fullsized":"2024-01-19T15:54:31.769038+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_clean_scrub_stamp":"2024-01-19T15:54:29.710724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:54:31.738395+0000","last_change":"2024-01-19T15:54:30.825327+0000","last_active":"2024-01-19T15:54:31.738395+0000","last_peered":"2024-01-19T15:54:31.738395+0000","last_clean":"2024-01-19T15:54:31.738395+0000","last_became_active":"2024-01-19T15:54:30.823434+0000","last_became_peered":"2024-01-19T15:54:30.823434+0000","last_unstale":"2024-01-19T15:54:31.738395+0000","last_undegraded":"2024-01-19T15:54:31.738395+0000","last_fullsized":"2024-01-19T15:54:31.738395+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_clean_scrub_stamp":"2024-01-19T15:54:29.710724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:54:31.736769+0000","last_change":"2024-01-19T15:54:30.805558+0000","last_active":"2024-01-19T15:54:31.736769+0000","last_peered":"2024-01-19T15:54:31.736769+0000","last_clean":"2024-01-19T15:54:31.736769+0000","last_became_active":"2024-01-19T15:54:30.803440+0000","last_became_peered":"2024-01-19T15:54:30.803440+0000","last_unstale":"2024-01-19T15:54:31.736769+0000","last_undegraded":"2024-01-19T15:54:31.736769+0000","last_fullsized":"2024-01-19T15:54:31.736769+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_clean_scrub_stamp":"2024-01-19T15:54:29.710724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:54:31.771920+0000","last_change":"2024-01-19T15:54:30.799497+0000","last_active":"2024-01-19T15:54:31.771920+0000","last_peered":"2024-01-19T15:54:31.771920+0000","last_clean":"2024-01-19T15:54:31.771920+0000","last_became_active":"2024-01-19T15:54:30.797796+0000","last_became_peered":"2024-01-19T15:54:30.797796+0000","last_unstale":"2024-01-19T15:54:31.771920+0000","last_undegraded":"2024-01-19T15:54:31.771920+0000","last_fullsized":"2024-01-19T15:54:31.771920+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_clean_scrub_stamp":"2024-01-19T15:54:29.710724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:54:31.767251+0000","last_change":"2024-01-19T15:54:29.786933+0000","last_active":"2024-01-19T15:54:31.767251+0000","last_peered":"2024-01-19T15:54:31.767251+0000","last_clean":"2024-01-19T15:54:31.767251+0000","last_became_active":"2024-01-19T15:54:29.778275+0000","last_became_peered":"2024-01-19T15:54:29.778275+0000","last_unstale":"2024-01-19T15:54:31.767251+0000","last_undegraded":"2024-01-19T15:54:31.767251+0000","last_fullsized":"2024-01-19T15:54:31.767251+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:54:28.710069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:54:28.710069+0000","last_clean_scrub_stamp":"2024-01-19T15:54:28.710069+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26988,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344852,"statfs":{"total":96636764160,"available":96609128448,"internally_reserved":0,"allocated":155648,"data_stored":611890,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26988,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344852,"statfs":{"total":96636764160,"available":96609128448,"internally_reserved":0,"allocated":155648,"data_stored":611890,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-19T15:54:35.197 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-19T15:54:35.302 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:35.308+0000 7f6d9c31e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:35.304 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:35.313+0000 7f6d9c31e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:35.479 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:54:35.480 INFO:teuthology.orchestra.run.smithi181.stderr:dumped all 2024-01-19T15:54:35.491 INFO:teuthology.orchestra.run.smithi181.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-01-19T15:54:33.702455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80900,"kb_used_data":404,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034620,"statfs":{"total":289910292480,"available":289827450880,"internally_reserved":0,"allocated":413696,"data_stored":1248195,"data_compressed":59922,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.948327"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:54:31.769615+0000","last_change":"2024-01-19T15:54:30.825728+0000","last_active":"2024-01-19T15:54:31.769615+0000","last_peered":"2024-01-19T15:54:31.769615+0000","last_clean":"2024-01-19T15:54:31.769615+0000","last_became_active":"2024-01-19T15:54:30.823983+0000","last_became_peered":"2024-01-19T15:54:30.823983+0000","last_unstale":"2024-01-19T15:54:31.769615+0000","last_undegraded":"2024-01-19T15:54:31.769615+0000","last_fullsized":"2024-01-19T15:54:31.769615+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_clean_scrub_stamp":"2024-01-19T15:54:29.710724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:54:31.764839+0000","last_change":"2024-01-19T15:54:30.837459+0000","last_active":"2024-01-19T15:54:31.764839+0000","last_peered":"2024-01-19T15:54:31.764839+0000","last_clean":"2024-01-19T15:54:31.764839+0000","last_became_active":"2024-01-19T15:54:30.835538+0000","last_became_peered":"2024-01-19T15:54:30.835538+0000","last_unstale":"2024-01-19T15:54:31.764839+0000","last_undegraded":"2024-01-19T15:54:31.764839+0000","last_fullsized":"2024-01-19T15:54:31.764839+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_clean_scrub_stamp":"2024-01-19T15:54:29.710724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:54:31.770478+0000","last_change":"2024-01-19T15:54:30.798448+0000","last_active":"2024-01-19T15:54:31.770478+0000","last_peered":"2024-01-19T15:54:31.770478+0000","last_clean":"2024-01-19T15:54:31.770478+0000","last_became_active":"2024-01-19T15:54:30.791023+0000","last_became_peered":"2024-01-19T15:54:30.791023+0000","last_unstale":"2024-01-19T15:54:31.770478+0000","last_undegraded":"2024-01-19T15:54:31.770478+0000","last_fullsized":"2024-01-19T15:54:31.770478+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_clean_scrub_stamp":"2024-01-19T15:54:29.710724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:54:31.766726+0000","last_change":"2024-01-19T15:54:30.840527+0000","last_active":"2024-01-19T15:54:31.766726+0000","last_peered":"2024-01-19T15:54:31.766726+0000","last_clean":"2024-01-19T15:54:31.766726+0000","last_became_active":"2024-01-19T15:54:30.836950+0000","last_became_peered":"2024-01-19T15:54:30.836950+0000","last_unstale":"2024-01-19T15:54:31.766726+0000","last_undegraded":"2024-01-19T15:54:31.766726+0000","last_fullsized":"2024-01-19T15:54:31.766726+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_clean_scrub_stamp":"2024-01-19T15:54:29.710724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:54:31.769038+0000","last_change":"2024-01-19T15:54:30.818720+0000","last_active":"2024-01-19T15:54:31.769038+0000","last_peered":"2024-01-19T15:54:31.769038+0000","last_clean":"2024-01-19T15:54:31.769038+0000","last_became_active":"2024-01-19T15:54:30.816411+0000","last_became_peered":"2024-01-19T15:54:30.816411+0000","last_unstale":"2024-01-19T15:54:31.769038+0000","last_undegraded":"2024-01-19T15:54:31.769038+0000","last_fullsized":"2024-01-19T15:54:31.769038+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_clean_scrub_stamp":"2024-01-19T15:54:29.710724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:54:31.738395+0000","last_change":"2024-01-19T15:54:30.825327+0000","last_active":"2024-01-19T15:54:31.738395+0000","last_peered":"2024-01-19T15:54:31.738395+0000","last_clean":"2024-01-19T15:54:31.738395+0000","last_became_active":"2024-01-19T15:54:30.823434+0000","last_became_peered":"2024-01-19T15:54:30.823434+0000","last_unstale":"2024-01-19T15:54:31.738395+0000","last_undegraded":"2024-01-19T15:54:31.738395+0000","last_fullsized":"2024-01-19T15:54:31.738395+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_clean_scrub_stamp":"2024-01-19T15:54:29.710724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:54:31.736769+0000","last_change":"2024-01-19T15:54:30.805558+0000","last_active":"2024-01-19T15:54:31.736769+0000","last_peered":"2024-01-19T15:54:31.736769+0000","last_clean":"2024-01-19T15:54:31.736769+0000","last_became_active":"2024-01-19T15:54:30.803440+0000","last_became_peered":"2024-01-19T15:54:30.803440+0000","last_unstale":"2024-01-19T15:54:31.736769+0000","last_undegraded":"2024-01-19T15:54:31.736769+0000","last_fullsized":"2024-01-19T15:54:31.736769+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_clean_scrub_stamp":"2024-01-19T15:54:29.710724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:54:31.771920+0000","last_change":"2024-01-19T15:54:30.799497+0000","last_active":"2024-01-19T15:54:31.771920+0000","last_peered":"2024-01-19T15:54:31.771920+0000","last_clean":"2024-01-19T15:54:31.771920+0000","last_became_active":"2024-01-19T15:54:30.797796+0000","last_became_peered":"2024-01-19T15:54:30.797796+0000","last_unstale":"2024-01-19T15:54:31.771920+0000","last_undegraded":"2024-01-19T15:54:31.771920+0000","last_fullsized":"2024-01-19T15:54:31.771920+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:54:29.710724+0000","last_clean_scrub_stamp":"2024-01-19T15:54:29.710724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":139,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-01-19T15:54:31.767251+0000","last_change":"2024-01-19T15:54:29.786933+0000","last_active":"2024-01-19T15:54:31.767251+0000","last_peered":"2024-01-19T15:54:31.767251+0000","last_clean":"2024-01-19T15:54:31.767251+0000","last_became_active":"2024-01-19T15:54:29.778275+0000","last_became_peered":"2024-01-19T15:54:29.778275+0000","last_unstale":"2024-01-19T15:54:31.767251+0000","last_undegraded":"2024-01-19T15:54:31.767251+0000","last_fullsized":"2024-01-19T15:54:31.767251+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-19T15:54:28.710069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-19T15:54:28.710069+0000","last_clean_scrub_stamp":"2024-01-19T15:54:28.710069+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":24415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26988,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344852,"statfs":{"total":96636764160,"available":96609128448,"internally_reserved":0,"allocated":155648,"data_stored":611890,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26988,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344852,"statfs":{"total":96636764160,"available":96609128448,"internally_reserved":0,"allocated":155648,"data_stored":611890,"data_compressed":29961,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[]}} 2024-01-19T15:54:35.492 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-19T15:54:35.492 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-19T15:54:35.492 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-19T15:54:35.492 DEBUG:teuthology.orchestra.run.smithi181:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-19T15:54:35.596 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:35.603+0000 7fda51e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:35.598 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:54:35.607+0000 7fda51e2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:54:35.811 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:54:35.812 INFO:teuthology.orchestra.run.smithi181.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-19T15:54:35.824 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-19T15:54:35.824 INFO:teuthology.run_tasks:Running task cbt... 2024-01-19T15:54:35.839 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi181.front.sepia.ceph.com', 'osds': ['ubuntu@smithi181.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi181.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi181.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi181.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randread'], 'norandommap': True, 'op_size': [4194304], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2], 'cmd_path': '/home/ubuntu/cephtest/fio/fio'}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-01-19T15:54:35.839 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-01-19T15:54:35.857 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-01-19T15:54:35.857 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-01-19T15:54:35.908 DEBUG:teuthology.orchestra.run.smithi181:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-01-19T15:54:35.963 INFO:teuthology.orchestra.run.smithi181.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-01-19T15:54:36.360 INFO:tasks.cbt:Installing collectl 2024-01-19T15:54:36.360 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-01-19T15:54:36.465 INFO:teuthology.orchestra.run.smithi181.stderr:--2024-01-19 15:54:36-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-01-19T15:54:36.612 INFO:teuthology.orchestra.run.smithi181.stderr:Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::ac40:9691, ... 2024-01-19T15:54:36.643 INFO:teuthology.orchestra.run.smithi181.stderr:Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. 2024-01-19T15:54:37.286 INFO:teuthology.orchestra.run.smithi181.stderr:HTTP request sent, awaiting response... 302 Found 2024-01-19T15:54:37.286 INFO:teuthology.orchestra.run.smithi181.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlqps8yP1TSn7He655TTDPo9OljY_hwLYS-a0vxoleuj6SXy_iy-SlqNK6ruEiSfzZrbAXUqFbZrKDwXlVTnsV5V2LGQ%3D%3D&use_mirror=cytranet&r= [following] 2024-01-19T15:54:37.286 INFO:teuthology.orchestra.run.smithi181.stderr:--2024-01-19 15:54:37-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABlqps8yP1TSn7He655TTDPo9OljY_hwLYS-a0vxoleuj6SXy_iy-SlqNK6ruEiSfzZrbAXUqFbZrKDwXlVTnsV5V2LGQ%3D%3D&use_mirror=cytranet&r= 2024-01-19T15:54:37.286 INFO:teuthology.orchestra.run.smithi181.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-01-19T15:54:37.357 INFO:teuthology.orchestra.run.smithi181.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-01-19T15:54:37.599 INFO:teuthology.orchestra.run.smithi181.stderr:HTTP request sent, awaiting response... 302 Found 2024-01-19T15:54:37.599 INFO:teuthology.orchestra.run.smithi181.stderr:Location: https://cytranet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2024-01-19T15:54:37.599 INFO:teuthology.orchestra.run.smithi181.stderr:--2024-01-19 15:54:37-- https://cytranet.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2024-01-19T15:54:37.610 INFO:teuthology.orchestra.run.smithi181.stderr:Resolving cytranet.dl.sourceforge.net (cytranet.dl.sourceforge.net)... 162.251.237.20 2024-01-19T15:54:37.681 INFO:teuthology.orchestra.run.smithi181.stderr:Connecting to cytranet.dl.sourceforge.net (cytranet.dl.sourceforge.net)|162.251.237.20|:443... connected. 2024-01-19T15:54:38.005 INFO:teuthology.orchestra.run.smithi181.stderr:HTTP request sent, awaiting response... 200 OK 2024-01-19T15:54:38.006 INFO:teuthology.orchestra.run.smithi181.stderr:Length: 620693 (606K) [application/x-gzip] 2024-01-19T15:54:38.006 INFO:teuthology.orchestra.run.smithi181.stderr:Saving to: ‘collectl.tar.gz’ 2024-01-19T15:54:38.006 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-01-19T15:54:38.096 INFO:teuthology.orchestra.run.smithi181.stderr: 0K .......... .......... .......... .......... .......... 8% 551K 1s 2024-01-19T15:54:38.165 INFO:teuthology.orchestra.run.smithi181.stderr: 50K .......... .......... .......... .......... .......... 16% 721K 1s 2024-01-19T15:54:38.167 INFO:teuthology.orchestra.run.smithi181.stderr: 100K .......... .......... .......... .......... .......... 24% 29.4M 0s 2024-01-19T15:54:38.217 INFO:teuthology.orchestra.run.smithi181.stderr: 150K .......... .......... .......... .......... .......... 32% 997K 0s 2024-01-19T15:54:38.238 INFO:teuthology.orchestra.run.smithi181.stderr: 200K .......... .......... .......... .......... .......... 41% 2.53M 0s 2024-01-19T15:54:38.238 INFO:teuthology.orchestra.run.smithi181.stderr: 250K .......... .......... .......... .......... .......... 49% 52.6M 0s 2024-01-19T15:54:38.238 INFO:teuthology.orchestra.run.smithi181.stderr: 300K .......... .......... .......... .......... .......... 57% 64.9M 0s 2024-01-19T15:54:38.287 INFO:teuthology.orchestra.run.smithi181.stderr: 350K .......... .......... .......... .......... .......... 65% 1.00M 0s 2024-01-19T15:54:38.306 INFO:teuthology.orchestra.run.smithi181.stderr: 400K .......... .......... .......... .......... .......... 74% 2.58M 0s 2024-01-19T15:54:38.308 INFO:teuthology.orchestra.run.smithi181.stderr: 450K .......... .......... .......... .......... .......... 82% 56.5M 0s 2024-01-19T15:54:38.308 INFO:teuthology.orchestra.run.smithi181.stderr: 500K .......... .......... .......... .......... .......... 90% 126M 0s 2024-01-19T15:54:38.308 INFO:teuthology.orchestra.run.smithi181.stderr: 550K .......... .......... .......... .......... .......... 98% 51.1M 0s 2024-01-19T15:54:38.308 INFO:teuthology.orchestra.run.smithi181.stderr: 600K ...... 100% 11.4T=0.3s 2024-01-19T15:54:38.308 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-01-19T15:54:38.308 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19 15:54:38 (1.96 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-01-19T15:54:38.308 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-01-19T15:54:38.343 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/ 2024-01-19T15:54:38.343 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/collectl.conf 2024-01-19T15:54:38.343 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/ 2024-01-19T15:54:38.343 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/SlowProc.html 2024-01-19T15:54:38.343 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-01-19T15:54:38.343 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/WhySummary.html 2024-01-19T15:54:38.344 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Socket.html 2024-01-19T15:54:38.344 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/colmux.html 2024-01-19T15:54:38.344 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Tutorial.html 2024-01-19T15:54:38.344 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Gexpr.html 2024-01-19T15:54:38.344 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Import.html 2024-01-19T15:54:38.344 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/style.css 2024-01-19T15:54:38.344 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-01-19T15:54:38.344 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-01-19T15:54:38.344 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/FileNaming.html 2024-01-19T15:54:38.344 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/TheMath.html 2024-01-19T15:54:38.344 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Environmental.html 2024-01-19T15:54:38.344 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Process.html 2024-01-19T15:54:38.344 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-01-19T15:54:38.344 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/HiResTime.html 2024-01-19T15:54:38.344 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-01-19T15:54:38.345 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/slub.jpg 2024-01-19T15:54:38.345 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Lustre.html 2024-01-19T15:54:38.345 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Documentation.html 2024-01-19T15:54:38.345 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Inodes.html 2024-01-19T15:54:38.345 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Graphite.html 2024-01-19T15:54:38.345 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/InputFiles.html 2024-01-19T15:54:38.345 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Export.html 2024-01-19T15:54:38.345 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Interrupts.html 2024-01-19T15:54:38.345 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Startup.html 2024-01-19T15:54:38.345 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Performance.html 2024-01-19T15:54:38.345 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Messages.html 2024-01-19T15:54:38.345 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Logging.html 2024-01-19T15:54:38.345 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Data-detail.html 2024-01-19T15:54:38.345 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/CPUs.html 2024-01-19T15:54:38.345 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Matrix.html 2024-01-19T15:54:38.346 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Data-brief.html 2024-01-19T15:54:38.346 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-01-19T15:54:38.346 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Misc.html 2024-01-19T15:54:38.346 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Examples.html 2024-01-19T15:54:38.346 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/OpenStack.html 2024-01-19T15:54:38.346 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-01-19T15:54:38.346 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-01-19T15:54:38.346 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-01-19T15:54:38.346 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-01-19T15:54:38.346 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Data.html 2024-01-19T15:54:38.346 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Exceptions.html 2024-01-19T15:54:38.346 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-01-19T15:54:38.346 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-01-19T15:54:38.346 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-01-19T15:54:38.346 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-01-19T15:54:38.347 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Playback.html 2024-01-19T15:54:38.347 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Hello.html 2024-01-19T15:54:38.347 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Sockets.html 2024-01-19T15:54:38.347 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Network.html 2024-01-19T15:54:38.347 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Infiniband.html 2024-01-19T15:54:38.347 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Memory.html 2024-01-19T15:54:38.347 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-01-19T15:54:38.347 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/Disks.html 2024-01-19T15:54:38.347 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-01-19T15:54:38.347 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/service/ 2024-01-19T15:54:38.347 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/service/collectl.service 2024-01-19T15:54:38.347 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/initd/ 2024-01-19T15:54:38.347 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/initd/collectl-suse 2024-01-19T15:54:38.347 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/initd/collectl-debian 2024-01-19T15:54:38.347 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/initd/collectl-generic 2024-01-19T15:54:38.348 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/initd/collectl 2024-01-19T15:54:38.348 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/misc.ph 2024-01-19T15:54:38.348 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/gexpr.ph 2024-01-19T15:54:38.348 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/statsd.ph 2024-01-19T15:54:38.348 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/envrules.std 2024-01-19T15:54:38.348 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/hello.ph 2024-01-19T15:54:38.348 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/vmsum.ph 2024-01-19T15:54:38.348 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/proctree.ph 2024-01-19T15:54:38.348 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/README-WINDOWS 2024-01-19T15:54:38.348 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/formatit.ph 2024-01-19T15:54:38.350 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/colmux 2024-01-19T15:54:38.350 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/INSTALL 2024-01-19T15:54:38.350 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/graphite.ph 2024-01-19T15:54:38.350 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/RELEASE-collectl 2024-01-19T15:54:38.350 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/vmstat.ph 2024-01-19T15:54:38.350 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/README 2024-01-19T15:54:38.350 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/man1/ 2024-01-19T15:54:38.350 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/man1/collectl.1 2024-01-19T15:54:38.351 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/man1/colmux.1 2024-01-19T15:54:38.351 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/lexpr.ph 2024-01-19T15:54:38.351 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/client.pl 2024-01-19T15:54:38.351 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/collectl 2024-01-19T15:54:38.351 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/COPYING 2024-01-19T15:54:38.351 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/vnet.ph 2024-01-19T15:54:38.351 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/GPL 2024-01-19T15:54:38.351 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/UNINSTALL 2024-01-19T15:54:38.351 INFO:teuthology.orchestra.run.smithi181.stdout:collectl-4.3.1/ARTISTIC 2024-01-19T15:54:38.404 INFO:teuthology.orchestra.run.smithi181.stderr:./INSTALL: line 90: chkconfig: command not found 2024-01-19T15:54:38.406 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-01-19T15:54:38.976 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:03:42 ago on Fri 19 Jan 2024 03:50:56 PM UTC. 2024-01-19T15:54:39.081 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-01-19T15:54:39.083 INFO:teuthology.orchestra.run.smithi181.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-01-19T15:54:39.139 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2024-01-19T15:54:39.140 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:54:39.140 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2024-01-19T15:54:39.140 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:54:39.140 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2024-01-19T15:54:39.140 INFO:teuthology.orchestra.run.smithi181.stdout: librbd-devel x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 23 k 2024-01-19T15:54:39.141 INFO:teuthology.orchestra.run.smithi181.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-01-19T15:54:39.141 INFO:teuthology.orchestra.run.smithi181.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-01-19T15:54:39.141 INFO:teuthology.orchestra.run.smithi181.stdout: perf x86_64 5.14.0-409.el9 CentOS-AppStream 8.0 M 2024-01-19T15:54:39.141 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2024-01-19T15:54:39.141 INFO:teuthology.orchestra.run.smithi181.stdout: libradospp-devel x86_64 2:19.0.0-767.gbdd6070c.el9 ceph 30 k 2024-01-19T15:54:39.141 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:54:39.141 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2024-01-19T15:54:39.141 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2024-01-19T15:54:39.141 INFO:teuthology.orchestra.run.smithi181.stdout:Install 5 Packages 2024-01-19T15:54:39.141 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:54:39.142 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 8.1 M 2024-01-19T15:54:39.142 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 11 M 2024-01-19T15:54:39.142 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2024-01-19T15:54:39.412 INFO:teuthology.orchestra.run.smithi181.stdout:(1/5): librbd-devel-19.0.0-767.gbdd6070c.el9.x8 123 kB/s | 23 kB 00:00 2024-01-19T15:54:39.538 INFO:teuthology.orchestra.run.smithi181.stdout:(2/5): libradospp-devel-19.0.0-767.gbdd6070c.el 97 kB/s | 30 kB 00:00 2024-01-19T15:54:39.738 INFO:teuthology.orchestra.run.smithi181.stdout:(3/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 66 kB/s | 13 kB 00:00 2024-01-19T15:54:39.855 INFO:teuthology.orchestra.run.smithi181.stdout:(4/5): pdsh-2.34-7.el9.x86_64.rpm 268 kB/s | 118 kB 00:00 2024-01-19T15:54:40.022 INFO:teuthology.orchestra.run.smithi181.stdout:(5/5): perf-5.14.0-409.el9.x86_64.rpm 10 MB/s | 8.0 MB 00:00 2024-01-19T15:54:40.024 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2024-01-19T15:54:40.024 INFO:teuthology.orchestra.run.smithi181.stdout:Total 9.2 MB/s | 8.1 MB 00:00 2024-01-19T15:54:40.024 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2024-01-19T15:54:40.044 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2024-01-19T15:54:40.045 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2024-01-19T15:54:40.203 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2024-01-19T15:54:40.203 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2024-01-19T15:54:40.596 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2024-01-19T15:54:40.658 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-01-19T15:54:40.688 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-01-19T15:54:40.710 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libradospp-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 3/5 2024-01-19T15:54:41.033 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librbd-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 4/5 2024-01-19T15:54:41.166 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : perf-5.14.0-409.el9.x86_64 5/5 2024-01-19T15:54:42.074 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: perf-5.14.0-409.el9.x86_64 5/5 2024-01-19T15:54:42.074 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libradospp-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 1/5 2024-01-19T15:54:42.074 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 2/5 2024-01-19T15:54:42.074 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : perf-5.14.0-409.el9.x86_64 3/5 2024-01-19T15:54:42.074 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-01-19T15:54:42.433 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-01-19T15:54:42.433 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:54:42.433 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2024-01-19T15:54:42.433 INFO:teuthology.orchestra.run.smithi181.stdout: libradospp-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:54:42.433 INFO:teuthology.orchestra.run.smithi181.stdout: librbd-devel-2:19.0.0-767.gbdd6070c.el9.x86_64 2024-01-19T15:54:42.433 INFO:teuthology.orchestra.run.smithi181.stdout: pdsh-2.34-7.el9.x86_64 2024-01-19T15:54:42.433 INFO:teuthology.orchestra.run.smithi181.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-01-19T15:54:42.433 INFO:teuthology.orchestra.run.smithi181.stdout: perf-5.14.0-409.el9.x86_64 2024-01-19T15:54:42.434 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:54:42.434 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2024-01-19T15:54:42.788 INFO:tasks.cbt:benchmark: librbdfio 2024-01-19T15:54:42.788 DEBUG:teuthology.orchestra.run.smithi181:> git clone -b master https://github.com/axboe/fio.git /home/ubuntu/cephtest/fio 2024-01-19T15:54:42.803 INFO:teuthology.orchestra.run.smithi181.stderr:Cloning into '/home/ubuntu/cephtest/fio'... 2024-01-19T15:54:44.846 DEBUG:teuthology.orchestra.run.smithi181:> cd /home/ubuntu/cephtest/fio && ./configure && make 2024-01-19T15:54:45.473 INFO:teuthology.orchestra.run.smithi181.stdout:Operating system Linux 2024-01-19T15:54:45.473 INFO:teuthology.orchestra.run.smithi181.stdout:CPU x86_64 2024-01-19T15:54:45.473 INFO:teuthology.orchestra.run.smithi181.stdout:Big endian no 2024-01-19T15:54:45.473 INFO:teuthology.orchestra.run.smithi181.stdout:Compiler gcc 2024-01-19T15:54:45.474 INFO:teuthology.orchestra.run.smithi181.stdout:Cross compile no 2024-01-19T15:54:45.474 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:54:45.474 INFO:teuthology.orchestra.run.smithi181.stdout:Static build no 2024-01-19T15:54:45.591 INFO:teuthology.orchestra.run.smithi181.stdout:Wordsize 64 2024-01-19T15:54:45.598 INFO:teuthology.orchestra.run.smithi181.stdout:zlib no 2024-01-19T15:54:45.663 INFO:teuthology.orchestra.run.smithi181.stdout:fcntl(F_FULLFSYNC) no 2024-01-19T15:54:45.677 INFO:teuthology.orchestra.run.smithi181.stdout:Linux AIO support no 2024-01-19T15:54:45.677 INFO:teuthology.orchestra.run.smithi181.stdout:Linux AIO support rw flags no 2024-01-19T15:54:45.737 INFO:teuthology.orchestra.run.smithi181.stdout:POSIX AIO support yes 2024-01-19T15:54:45.737 INFO:teuthology.orchestra.run.smithi181.stdout:POSIX AIO support needs -lrt no 2024-01-19T15:54:45.767 INFO:teuthology.orchestra.run.smithi181.stdout:POSIX AIO fsync yes 2024-01-19T15:54:45.799 INFO:teuthology.orchestra.run.smithi181.stdout:POSIX pshared support yes 2024-01-19T15:54:45.843 INFO:teuthology.orchestra.run.smithi181.stdout:pthread_condattr_setclock() yes 2024-01-19T15:54:45.882 INFO:teuthology.orchestra.run.smithi181.stdout:pthread_sigmask() yes 2024-01-19T15:54:45.917 INFO:teuthology.orchestra.run.smithi181.stdout:pthread_getaffinity_np() yes 2024-01-19T15:54:45.927 INFO:teuthology.orchestra.run.smithi181.stdout:Solaris AIO support no 2024-01-19T15:54:45.973 INFO:teuthology.orchestra.run.smithi181.stdout:__sync_fetch_and_add yes 2024-01-19T15:54:46.038 INFO:teuthology.orchestra.run.smithi181.stdout:__sync_synchronize yes 2024-01-19T15:54:46.067 INFO:teuthology.orchestra.run.smithi181.stdout:__sync_val_compare_and_swap yes 2024-01-19T15:54:46.075 INFO:teuthology.orchestra.run.smithi181.stdout:libverbs no 2024-01-19T15:54:46.105 INFO:teuthology.orchestra.run.smithi181.stdout:rdmacm no 2024-01-19T15:54:46.113 INFO:teuthology.orchestra.run.smithi181.stdout:librpma no 2024-01-19T15:54:46.123 INFO:teuthology.orchestra.run.smithi181.stdout:libprotobuf_c no 2024-01-19T15:54:46.153 INFO:teuthology.orchestra.run.smithi181.stdout:asprintf() yes 2024-01-19T15:54:46.184 INFO:teuthology.orchestra.run.smithi181.stdout:vasprintf() yes 2024-01-19T15:54:46.215 INFO:teuthology.orchestra.run.smithi181.stdout:Linux fallocate yes 2024-01-19T15:54:46.247 INFO:teuthology.orchestra.run.smithi181.stdout:POSIX fadvise yes 2024-01-19T15:54:46.279 INFO:teuthology.orchestra.run.smithi181.stdout:POSIX fallocate yes 2024-01-19T15:54:46.309 INFO:teuthology.orchestra.run.smithi181.stdout:sched_setaffinity(3 arg) yes 2024-01-19T15:54:46.309 INFO:teuthology.orchestra.run.smithi181.stdout:sched_setaffinity(2 arg) no 2024-01-19T15:54:46.340 INFO:teuthology.orchestra.run.smithi181.stdout:clock_gettime yes 2024-01-19T15:54:46.371 INFO:teuthology.orchestra.run.smithi181.stdout:CLOCK_MONOTONIC yes 2024-01-19T15:54:46.410 INFO:teuthology.orchestra.run.smithi181.stdout:clockid_t yes 2024-01-19T15:54:46.448 INFO:teuthology.orchestra.run.smithi181.stdout:gettimeofday yes 2024-01-19T15:54:46.480 INFO:teuthology.orchestra.run.smithi181.stdout:fdatasync yes 2024-01-19T15:54:46.511 INFO:teuthology.orchestra.run.smithi181.stdout:pipe() yes 2024-01-19T15:54:46.541 INFO:teuthology.orchestra.run.smithi181.stdout:pipe2() yes 2024-01-19T15:54:46.571 INFO:teuthology.orchestra.run.smithi181.stdout:pread() yes 2024-01-19T15:54:46.644 INFO:teuthology.orchestra.run.smithi181.stdout:sync_file_range yes 2024-01-19T15:54:46.651 INFO:teuthology.orchestra.run.smithi181.stdout:ASharedMemory_create no 2024-01-19T15:54:46.663 INFO:teuthology.orchestra.run.smithi181.stdout:EXT4 move extent yes 2024-01-19T15:54:46.694 INFO:teuthology.orchestra.run.smithi181.stdout:Linux splice(2) yes 2024-01-19T15:54:46.702 INFO:teuthology.orchestra.run.smithi181.stdout:libnuma no 2024-01-19T15:54:46.731 INFO:teuthology.orchestra.run.smithi181.stdout:strsep yes 2024-01-19T15:54:46.761 INFO:teuthology.orchestra.run.smithi181.stdout:strcasestr yes 2024-01-19T15:54:46.774 INFO:teuthology.orchestra.run.smithi181.stdout:strlcat no 2024-01-19T15:54:46.809 INFO:teuthology.orchestra.run.smithi181.stdout:getopt_long_only() yes 2024-01-19T15:54:46.872 INFO:teuthology.orchestra.run.smithi181.stdout:inet_aton yes 2024-01-19T15:54:46.904 INFO:teuthology.orchestra.run.smithi181.stdout:socklen_t yes 2024-01-19T15:54:46.943 INFO:teuthology.orchestra.run.smithi181.stdout:__thread yes 2024-01-19T15:54:46.973 INFO:teuthology.orchestra.run.smithi181.stdout:RUSAGE_THREAD yes 2024-01-19T15:54:47.002 INFO:teuthology.orchestra.run.smithi181.stdout:SCHED_IDLE yes 2024-01-19T15:54:47.036 INFO:teuthology.orchestra.run.smithi181.stdout:TCP_NODELAY yes 2024-01-19T15:54:47.069 INFO:teuthology.orchestra.run.smithi181.stdout:Net engine window_size yes 2024-01-19T15:54:47.104 INFO:teuthology.orchestra.run.smithi181.stdout:TCP_MAXSEG yes 2024-01-19T15:54:47.133 INFO:teuthology.orchestra.run.smithi181.stdout:RLIMIT_MEMLOCK yes 2024-01-19T15:54:47.165 INFO:teuthology.orchestra.run.smithi181.stdout:pwritev/preadv yes 2024-01-19T15:54:47.197 INFO:teuthology.orchestra.run.smithi181.stdout:pwritev2/preadv2 yes 2024-01-19T15:54:47.235 INFO:teuthology.orchestra.run.smithi181.stdout:IPv6 helpers yes 2024-01-19T15:54:47.251 INFO:teuthology.orchestra.run.smithi181.stdout:http engine no 2024-01-19T15:54:47.342 INFO:teuthology.orchestra.run.smithi181.stdout:Rados engine yes 2024-01-19T15:54:47.446 INFO:teuthology.orchestra.run.smithi181.stdout:Rados Block Device engine yes 2024-01-19T15:54:47.551 INFO:teuthology.orchestra.run.smithi181.stdout:rbd_poll yes 2024-01-19T15:54:47.657 INFO:teuthology.orchestra.run.smithi181.stdout:rbd_invalidate_cache yes 2024-01-19T15:54:47.687 INFO:teuthology.orchestra.run.smithi181.stdout:setvbuf yes 2024-01-19T15:54:47.695 INFO:teuthology.orchestra.run.smithi181.stdout:Gluster API engine no 2024-01-19T15:54:47.710 INFO:teuthology.orchestra.run.smithi181.stdout:s390_z196_facilities no 2024-01-19T15:54:47.710 INFO:teuthology.orchestra.run.smithi181.stdout:HDFS engine no 2024-01-19T15:54:47.755 INFO:teuthology.orchestra.run.smithi181.stdout:MTD yes 2024-01-19T15:54:47.762 INFO:teuthology.orchestra.run.smithi181.stdout:libpmem no 2024-01-19T15:54:47.763 INFO:teuthology.orchestra.run.smithi181.stdout:libpmem1_5 no 2024-01-19T15:54:47.770 INFO:teuthology.orchestra.run.smithi181.stdout:libpmem2 no 2024-01-19T15:54:47.770 INFO:teuthology.orchestra.run.smithi181.stdout:PMDK dev-dax engine no 2024-01-19T15:54:47.770 INFO:teuthology.orchestra.run.smithi181.stdout:PMDK libpmem engine no 2024-01-19T15:54:47.777 INFO:teuthology.orchestra.run.smithi181.stdout:DDN's Infinite Memory Engine no 2024-01-19T15:54:47.777 INFO:teuthology.orchestra.run.smithi181.stdout:iscsi engine no 2024-01-19T15:54:47.777 INFO:teuthology.orchestra.run.smithi181.stdout:NBD engine no 2024-01-19T15:54:47.785 INFO:teuthology.orchestra.run.smithi181.stdout:DAOS File System (dfs) Engine no 2024-01-19T15:54:47.794 INFO:teuthology.orchestra.run.smithi181.stdout:NFS engine no 2024-01-19T15:54:47.794 INFO:teuthology.orchestra.run.smithi181.stdout:lex/yacc for arithmetic no 2024-01-19T15:54:47.832 INFO:teuthology.orchestra.run.smithi181.stdout:getmntent yes 2024-01-19T15:54:47.859 INFO:teuthology.orchestra.run.smithi181.stdout:getmntinfo no 2024-01-19T15:54:47.913 INFO:teuthology.orchestra.run.smithi181.stdout:Static Assert yes 2024-01-19T15:54:47.940 INFO:teuthology.orchestra.run.smithi181.stdout:bool yes 2024-01-19T15:54:47.975 INFO:teuthology.orchestra.run.smithi181.stdout:strndup yes 2024-01-19T15:54:47.981 INFO:teuthology.orchestra.run.smithi181.stdout:Valgrind headers no 2024-01-19T15:54:48.017 INFO:teuthology.orchestra.run.smithi181.stdout:Zoned block device support yes 2024-01-19T15:54:48.044 INFO:teuthology.orchestra.run.smithi181.stdout:Zoned block device capacity yes 2024-01-19T15:54:48.051 INFO:teuthology.orchestra.run.smithi181.stdout:libzbc engine no 2024-01-19T15:54:48.079 INFO:teuthology.orchestra.run.smithi181.stdout:NVMe uring command support yes 2024-01-19T15:54:48.093 INFO:teuthology.orchestra.run.smithi181.stdout:xnvme engine no 2024-01-19T15:54:48.101 INFO:teuthology.orchestra.run.smithi181.stdout:isal no 2024-01-19T15:54:48.115 INFO:teuthology.orchestra.run.smithi181.stdout:libblkio engine no 2024-01-19T15:54:48.115 INFO:teuthology.orchestra.run.smithi181.stdout:march_armv8_a_crc_crypto no 2024-01-19T15:54:48.115 INFO:teuthology.orchestra.run.smithi181.stdout:cuda no 2024-01-19T15:54:48.115 INFO:teuthology.orchestra.run.smithi181.stdout:libcufile no 2024-01-19T15:54:48.142 INFO:teuthology.orchestra.run.smithi181.stdout:Build march=native yes 2024-01-19T15:54:48.150 INFO:teuthology.orchestra.run.smithi181.stdout:CUnit no 2024-01-19T15:54:48.179 INFO:teuthology.orchestra.run.smithi181.stdout:__kernel_rwf_t yes 2024-01-19T15:54:48.206 INFO:teuthology.orchestra.run.smithi181.stdout:-Wimplicit-fallthrough=2 yes 2024-01-19T15:54:48.236 INFO:teuthology.orchestra.run.smithi181.stdout:-Wno-stringop-truncation yes 2024-01-19T15:54:48.266 INFO:teuthology.orchestra.run.smithi181.stdout:MADV_HUGEPAGE yes 2024-01-19T15:54:48.296 INFO:teuthology.orchestra.run.smithi181.stdout:gettid yes 2024-01-19T15:54:48.331 INFO:teuthology.orchestra.run.smithi181.stdout:statx(2)/libc yes 2024-01-19T15:54:48.366 INFO:teuthology.orchestra.run.smithi181.stdout:statx(2)/syscall yes 2024-01-19T15:54:48.368 INFO:teuthology.orchestra.run.smithi181.stdout:Windows PDB generation no 2024-01-19T15:54:48.398 INFO:teuthology.orchestra.run.smithi181.stdout:timerfd_create yes 2024-01-19T15:54:48.400 INFO:teuthology.orchestra.run.smithi181.stdout:Consider installing zlib1g-dev (zlib-devel) as some fio features depend on it. 2024-01-19T15:54:48.400 INFO:teuthology.orchestra.run.smithi181.stdout:Lib-based ioengines dynamic no 2024-01-19T15:54:48.462 INFO:teuthology.orchestra.run.smithi181.stdout:TCMalloc support yes 2024-01-19T15:54:48.463 INFO:teuthology.orchestra.run.smithi181.stdout:seed_buckets 4 2024-01-19T15:54:48.544 INFO:teuthology.orchestra.run.smithi181.stderr:FIO_VERSION = fio-3.36-42-gaa84 2024-01-19T15:54:48.576 INFO:teuthology.orchestra.run.smithi181.stdout: CC crc/crc16.o 2024-01-19T15:54:48.667 INFO:teuthology.orchestra.run.smithi181.stdout: CC crc/crc32.o 2024-01-19T15:54:48.703 INFO:teuthology.orchestra.run.smithi181.stdout: CC crc/crc32c-arm64.o 2024-01-19T15:54:48.778 INFO:teuthology.orchestra.run.smithi181.stdout: CC crc/crc32c-intel.o 2024-01-19T15:54:48.822 INFO:teuthology.orchestra.run.smithi181.stdout: CC crc/crc32c.o 2024-01-19T15:54:48.861 INFO:teuthology.orchestra.run.smithi181.stdout: CC crc/crc64.o 2024-01-19T15:54:48.899 INFO:teuthology.orchestra.run.smithi181.stdout: CC crc/crc7.o 2024-01-19T15:54:48.932 INFO:teuthology.orchestra.run.smithi181.stdout: CC crc/crct10dif_common.o 2024-01-19T15:54:48.966 INFO:teuthology.orchestra.run.smithi181.stdout: CC crc/fnv.o 2024-01-19T15:54:49.006 INFO:teuthology.orchestra.run.smithi181.stdout: CC crc/md5.o 2024-01-19T15:54:49.108 INFO:teuthology.orchestra.run.smithi181.stdout: CC crc/murmur3.o 2024-01-19T15:54:49.150 INFO:teuthology.orchestra.run.smithi181.stdout: CC crc/sha1.o 2024-01-19T15:54:49.344 INFO:teuthology.orchestra.run.smithi181.stdout: CC crc/sha256.o 2024-01-19T15:54:49.637 INFO:teuthology.orchestra.run.smithi181.stdout: CC crc/sha3.o 2024-01-19T15:54:49.782 INFO:teuthology.orchestra.run.smithi181.stdout: CC crc/sha512.o 2024-01-19T15:54:49.889 INFO:teuthology.orchestra.run.smithi181.stdout: CC crc/test.o 2024-01-19T15:54:50.050 INFO:teuthology.orchestra.run.smithi181.stdout: CC crc/xxhash.o 2024-01-19T15:54:50.144 INFO:teuthology.orchestra.run.smithi181.stdout: CC lib/axmap.o 2024-01-19T15:54:50.265 INFO:teuthology.orchestra.run.smithi181.stdout: CC lib/bloom.o 2024-01-19T15:54:50.347 INFO:teuthology.orchestra.run.smithi181.stdout: CC lib/flist_sort.o 2024-01-19T15:54:50.418 INFO:teuthology.orchestra.run.smithi181.stdout: CC lib/gauss.o 2024-01-19T15:54:50.514 INFO:teuthology.orchestra.run.smithi181.stdout: CC lib/getrusage.o 2024-01-19T15:54:50.549 INFO:teuthology.orchestra.run.smithi181.stdout: CC lib/hweight.o 2024-01-19T15:54:50.588 INFO:teuthology.orchestra.run.smithi181.stdout: CC lib/ieee754.o 2024-01-19T15:54:50.636 INFO:teuthology.orchestra.run.smithi181.stdout: CC lib/lfsr.o 2024-01-19T15:54:50.735 INFO:teuthology.orchestra.run.smithi181.stdout: CC lib/memalign.o 2024-01-19T15:54:50.778 INFO:teuthology.orchestra.run.smithi181.stdout: CC lib/memcpy.o 2024-01-19T15:54:50.920 INFO:teuthology.orchestra.run.smithi181.stdout: CC lib/mountcheck.o 2024-01-19T15:54:50.961 INFO:teuthology.orchestra.run.smithi181.stdout: CC lib/num2str.o 2024-01-19T15:54:51.026 INFO:teuthology.orchestra.run.smithi181.stdout: CC lib/output_buffer.o 2024-01-19T15:54:51.073 INFO:teuthology.orchestra.run.smithi181.stdout: CC lib/pattern.o 2024-01-19T15:54:51.217 INFO:teuthology.orchestra.run.smithi181.stdout: CC lib/prio_tree.o 2024-01-19T15:54:51.335 INFO:teuthology.orchestra.run.smithi181.stdout: CC lib/rand.o 2024-01-19T15:54:51.522 INFO:teuthology.orchestra.run.smithi181.stdout: CC lib/rbtree.o 2024-01-19T15:54:51.643 INFO:teuthology.orchestra.run.smithi181.stdout: CC lib/strntol.o 2024-01-19T15:54:51.691 INFO:teuthology.orchestra.run.smithi181.stdout: CC lib/zipf.o 2024-01-19T15:54:51.811 INFO:teuthology.orchestra.run.smithi181.stdout: CC gettime.o 2024-01-19T15:54:52.095 INFO:teuthology.orchestra.run.smithi181.stdout: CC ioengines.o 2024-01-19T15:54:52.356 INFO:teuthology.orchestra.run.smithi181.stdout: CC init.o 2024-01-19T15:54:53.097 INFO:teuthology.orchestra.run.smithi181.stdout: CC stat.o 2024-01-19T15:54:54.964 INFO:teuthology.orchestra.run.smithi181.stdout: CC log.o 2024-01-19T15:54:55.109 INFO:teuthology.orchestra.run.smithi181.stdout: CC time.o 2024-01-19T15:54:55.231 INFO:teuthology.orchestra.run.smithi181.stdout: CC filesetup.o 2024-01-19T15:54:55.919 INFO:teuthology.orchestra.run.smithi181.stdout: CC eta.o 2024-01-19T15:54:56.229 INFO:teuthology.orchestra.run.smithi181.stdout: CC verify.o 2024-01-19T15:54:56.901 INFO:teuthology.orchestra.run.smithi181.stdout: CC memory.o 2024-01-19T15:54:57.070 INFO:teuthology.orchestra.run.smithi181.stdout: CC io_u.o 2024-01-19T15:54:57.817 INFO:teuthology.orchestra.run.smithi181.stdout: CC parse.o 2024-01-19T15:54:58.270 INFO:teuthology.orchestra.run.smithi181.stdout: CC fio_sem.o 2024-01-19T15:54:58.372 INFO:teuthology.orchestra.run.smithi181.stdout: CC rwlock.o 2024-01-19T15:54:58.455 INFO:teuthology.orchestra.run.smithi181.stdout: CC pshared.o 2024-01-19T15:54:58.519 INFO:teuthology.orchestra.run.smithi181.stdout: CC options.o 2024-01-19T15:54:59.292 INFO:teuthology.orchestra.run.smithi181.stdout: CC smalloc.o 2024-01-19T15:54:59.472 INFO:teuthology.orchestra.run.smithi181.stdout: CC filehash.o 2024-01-19T15:54:59.624 INFO:teuthology.orchestra.run.smithi181.stdout: CC profile.o 2024-01-19T15:54:59.739 INFO:teuthology.orchestra.run.smithi181.stdout: CC debug.o 2024-01-19T15:54:59.785 INFO:teuthology.orchestra.run.smithi181.stdout: CC engines/cpu.o 2024-01-19T15:54:59.923 INFO:teuthology.orchestra.run.smithi181.stdout: CC engines/mmap.o 2024-01-19T15:55:00.087 INFO:teuthology.orchestra.run.smithi181.stdout: CC engines/sync.o 2024-01-19T15:55:00.308 INFO:teuthology.orchestra.run.smithi181.stdout: CC engines/null.o 2024-01-19T15:55:00.422 INFO:teuthology.orchestra.run.smithi181.stdout: CC engines/net.o 2024-01-19T15:55:00.828 INFO:teuthology.orchestra.run.smithi181.stdout: CC engines/ftruncate.o 2024-01-19T15:55:00.926 INFO:teuthology.orchestra.run.smithi181.stdout: CC engines/fileoperations.o 2024-01-19T15:55:01.066 INFO:teuthology.orchestra.run.smithi181.stdout: CC engines/exec.o 2024-01-19T15:55:01.217 INFO:teuthology.orchestra.run.smithi181.stdout: CC server.o 2024-01-19T15:55:01.948 INFO:teuthology.orchestra.run.smithi181.stdout: CC client.o 2024-01-19T15:55:02.475 INFO:teuthology.orchestra.run.smithi181.stdout: CC iolog.o 2024-01-19T15:55:02.915 INFO:teuthology.orchestra.run.smithi181.stdout: CC backend.o 2024-01-19T15:55:03.740 INFO:teuthology.orchestra.run.smithi181.stdout: CC libfio.o 2024-01-19T15:55:03.916 INFO:teuthology.orchestra.run.smithi181.stdout: CC flow.o 2024-01-19T15:55:04.031 INFO:teuthology.orchestra.run.smithi181.stdout: CC cconv.o 2024-01-19T15:55:04.379 INFO:teuthology.orchestra.run.smithi181.stdout: CC gettime-thread.o 2024-01-19T15:55:04.480 INFO:teuthology.orchestra.run.smithi181.stdout: CC helpers.o 2024-01-19T15:55:04.540 INFO:teuthology.orchestra.run.smithi181.stdout: CC json.o 2024-01-19T15:55:04.720 INFO:teuthology.orchestra.run.smithi181.stdout: CC idletime.o 2024-01-19T15:55:04.938 INFO:teuthology.orchestra.run.smithi181.stdout: CC td_error.o 2024-01-19T15:55:05.037 INFO:teuthology.orchestra.run.smithi181.stdout: CC profiles/tiobench.o 2024-01-19T15:55:05.131 INFO:teuthology.orchestra.run.smithi181.stdout: CC profiles/act.o 2024-01-19T15:55:05.289 INFO:teuthology.orchestra.run.smithi181.stdout: CC io_u_queue.o 2024-01-19T15:55:05.338 INFO:teuthology.orchestra.run.smithi181.stdout: CC filelock.o 2024-01-19T15:55:05.463 INFO:teuthology.orchestra.run.smithi181.stdout: CC workqueue.o 2024-01-19T15:55:05.634 INFO:teuthology.orchestra.run.smithi181.stdout: CC rate-submit.o 2024-01-19T15:55:05.782 INFO:teuthology.orchestra.run.smithi181.stdout: CC optgroup.o 2024-01-19T15:55:05.827 INFO:teuthology.orchestra.run.smithi181.stdout: CC helper_thread.o 2024-01-19T15:55:06.004 INFO:teuthology.orchestra.run.smithi181.stdout: CC steadystate.o 2024-01-19T15:55:06.234 INFO:teuthology.orchestra.run.smithi181.stdout: CC zone-dist.o 2024-01-19T15:55:06.368 INFO:teuthology.orchestra.run.smithi181.stdout: CC zbd.o 2024-01-19T15:55:07.050 INFO:teuthology.orchestra.run.smithi181.stdout: CC dedupe.o 2024-01-19T15:55:07.193 INFO:teuthology.orchestra.run.smithi181.stdout: CC fdp.o 2024-01-19T15:55:07.329 INFO:teuthology.orchestra.run.smithi181.stdout: CC engines/posixaio.o 2024-01-19T15:55:07.461 INFO:teuthology.orchestra.run.smithi181.stdout: CC engines/falloc.o 2024-01-19T15:55:07.569 INFO:teuthology.orchestra.run.smithi181.stdout: CC engines/e4defrag.o 2024-01-19T15:55:07.697 INFO:teuthology.orchestra.run.smithi181.stdout: CC engines/splice.o 2024-01-19T15:55:07.886 INFO:teuthology.orchestra.run.smithi181.stdout: CC oslib/asprintf.o 2024-01-19T15:55:07.921 INFO:teuthology.orchestra.run.smithi181.stdout: CC oslib/strlcat.o 2024-01-19T15:55:07.964 INFO:teuthology.orchestra.run.smithi181.stdout: CC engines/mtd.o 2024-01-19T15:55:08.122 INFO:teuthology.orchestra.run.smithi181.stdout: CC oslib/libmtd.o 2024-01-19T15:55:08.509 INFO:teuthology.orchestra.run.smithi181.stdout: CC oslib/libmtd_legacy.o 2024-01-19T15:55:08.638 INFO:teuthology.orchestra.run.smithi181.stdout: CC diskutil.o 2024-01-19T15:55:08.845 INFO:teuthology.orchestra.run.smithi181.stdout: CC fifo.o 2024-01-19T15:55:08.901 INFO:teuthology.orchestra.run.smithi181.stdout: CC blktrace.o 2024-01-19T15:55:09.225 INFO:teuthology.orchestra.run.smithi181.stdout: CC cgroup.o 2024-01-19T15:55:09.374 INFO:teuthology.orchestra.run.smithi181.stdout: CC trim.o 2024-01-19T15:55:09.490 INFO:teuthology.orchestra.run.smithi181.stdout: CC engines/sg.o 2024-01-19T15:55:09.872 INFO:teuthology.orchestra.run.smithi181.stdout: CC oslib/linux-dev-lookup.o 2024-01-19T15:55:09.928 INFO:teuthology.orchestra.run.smithi181.stdout: CC engines/io_uring.o 2024-01-19T15:55:10.302 INFO:teuthology.orchestra.run.smithi181.stdout: CC engines/nvme.o 2024-01-19T15:55:10.584 INFO:teuthology.orchestra.run.smithi181.stdout: CC oslib/linux-blkzoned.o 2024-01-19T15:55:10.733 INFO:teuthology.orchestra.run.smithi181.stdout: CC engines/cmdprio.o 2024-01-19T15:55:10.973 INFO:teuthology.orchestra.run.smithi181.stdout: CC engines/rados.o 2024-01-19T15:55:11.142 INFO:teuthology.orchestra.run.smithi181.stdout: CC engines/rbd.o 2024-01-19T15:55:11.320 INFO:teuthology.orchestra.run.smithi181.stdout: CC fio.o 2024-01-19T15:55:11.407 INFO:teuthology.orchestra.run.smithi181.stdout: LINK fio 2024-01-19T15:55:11.596 INFO:teuthology.orchestra.run.smithi181.stdout: CC t/log.o 2024-01-19T15:55:11.637 INFO:teuthology.orchestra.run.smithi181.stdout: CC t/genzipf.o 2024-01-19T15:55:11.776 INFO:teuthology.orchestra.run.smithi181.stdout: CC oslib/strcasestr.o 2024-01-19T15:55:11.800 INFO:teuthology.orchestra.run.smithi181.stdout: CC oslib/strndup.o 2024-01-19T15:55:11.823 INFO:teuthology.orchestra.run.smithi181.stdout: LINK t/fio-genzipf 2024-01-19T15:55:11.911 INFO:teuthology.orchestra.run.smithi181.stdout: CC t/btrace2fio.o 2024-01-19T15:55:12.327 INFO:teuthology.orchestra.run.smithi181.stdout: LINK t/fio-btrace2fio 2024-01-19T15:55:12.416 INFO:teuthology.orchestra.run.smithi181.stdout: CC t/verify-state.o 2024-01-19T15:55:12.515 INFO:teuthology.orchestra.run.smithi181.stdout: CC t/debug.o 2024-01-19T15:55:12.550 INFO:teuthology.orchestra.run.smithi181.stdout: LINK t/fio-verify-state 2024-01-19T15:55:12.637 INFO:teuthology.orchestra.run.smithi181.stdout: CC t/stest.o 2024-01-19T15:55:12.707 INFO:teuthology.orchestra.run.smithi181.stdout: CC t/arch.o 2024-01-19T15:55:12.736 INFO:teuthology.orchestra.run.smithi181.stdout: LINK t/stest 2024-01-19T15:55:12.824 INFO:teuthology.orchestra.run.smithi181.stdout: CC t/ieee754.o 2024-01-19T15:55:12.863 INFO:teuthology.orchestra.run.smithi181.stdout: LINK t/ieee754 2024-01-19T15:55:12.949 INFO:teuthology.orchestra.run.smithi181.stdout: CC t/axmap.o 2024-01-19T15:55:13.047 INFO:teuthology.orchestra.run.smithi181.stdout: LINK t/axmap 2024-01-19T15:55:13.135 INFO:teuthology.orchestra.run.smithi181.stdout: CC t/lfsr-test.o 2024-01-19T15:55:13.233 INFO:teuthology.orchestra.run.smithi181.stdout: LINK t/lfsr-test 2024-01-19T15:55:13.325 INFO:teuthology.orchestra.run.smithi181.stdout: CC t/gen-rand.o 2024-01-19T15:55:13.420 INFO:teuthology.orchestra.run.smithi181.stdout: LINK t/gen-rand 2024-01-19T15:55:13.510 INFO:teuthology.orchestra.run.smithi181.stdout: CC t/memlock.o 2024-01-19T15:55:13.570 INFO:teuthology.orchestra.run.smithi181.stdout: LINK t/memlock 2024-01-19T15:55:13.656 INFO:teuthology.orchestra.run.smithi181.stdout: CC t/read-to-pipe-async.o 2024-01-19T15:55:13.844 INFO:teuthology.orchestra.run.smithi181.stdout: LINK t/read-to-pipe-async 2024-01-19T15:55:13.932 INFO:teuthology.orchestra.run.smithi181.stdout: CC t/io_uring.o 2024-01-19T15:55:14.466 INFO:teuthology.orchestra.run.smithi181.stdout: LINK t/io_uring 2024-01-19T15:55:14.556 DEBUG:teuthology.orchestra.run.smithi181:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-01-19T15:55:14.584 DEBUG:teuthology.orchestra.run.smithi181:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-01-19T15:55:14.831 INFO:teuthology.orchestra.run.smithi181.stderr:15:55:14 - DEBUG - cbt - Settings.cluster: 2024-01-19T15:55:14.831 INFO:teuthology.orchestra.run.smithi181.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-01-19T15:55:14.831 INFO:teuthology.orchestra.run.smithi181.stderr: 'clients': ['ubuntu@smithi181.front.sepia.ceph.com'], 2024-01-19T15:55:14.831 INFO:teuthology.orchestra.run.smithi181.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-01-19T15:55:14.831 INFO:teuthology.orchestra.run.smithi181.stderr: 'head': 'ubuntu@smithi181.front.sepia.ceph.com', 2024-01-19T15:55:14.832 INFO:teuthology.orchestra.run.smithi181.stderr: 'is_teuthology': True, 2024-01-19T15:55:14.832 INFO:teuthology.orchestra.run.smithi181.stderr: 'iterations': 1, 2024-01-19T15:55:14.832 INFO:teuthology.orchestra.run.smithi181.stderr: 'mons': ['ubuntu@smithi181.front.sepia.ceph.com'], 2024-01-19T15:55:14.832 INFO:teuthology.orchestra.run.smithi181.stderr: 'osds': ['ubuntu@smithi181.front.sepia.ceph.com'], 2024-01-19T15:55:14.832 INFO:teuthology.orchestra.run.smithi181.stderr: 'osds_per_node': 3, 2024-01-19T15:55:14.832 INFO:teuthology.orchestra.run.smithi181.stderr: 'pid_dir': '/var/run/ceph', 2024-01-19T15:55:14.832 INFO:teuthology.orchestra.run.smithi181.stderr: 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 2024-01-19T15:55:14.832 INFO:teuthology.orchestra.run.smithi181.stderr: 'rebuild_every_test': False, 2024-01-19T15:55:14.832 INFO:teuthology.orchestra.run.smithi181.stderr: 'rgws': {'ubuntu@smithi181.front.sepia.ceph.com': None}, 2024-01-19T15:55:14.832 INFO:teuthology.orchestra.run.smithi181.stderr: 'tmp_dir': '/tmp/cbt', 2024-01-19T15:55:14.832 INFO:teuthology.orchestra.run.smithi181.stderr: 'use_existing': True, 2024-01-19T15:55:14.832 INFO:teuthology.orchestra.run.smithi181.stderr: 'user': 'ubuntu'} 2024-01-19T15:55:14.833 INFO:teuthology.orchestra.run.smithi181.stderr:15:55:14 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:55:14.833 INFO:teuthology.orchestra.run.smithi181.stderr:15:55:14 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:55:14.833 INFO:teuthology.orchestra.run.smithi181.stderr:15:55:14 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:55:14.833 INFO:teuthology.orchestra.run.smithi181.stderr:15:55:14 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread; fi 2024-01-19T15:55:14.835 INFO:teuthology.orchestra.run.smithi181.stderr:15:55:14 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:55:14.835 INFO:teuthology.orchestra.run.smithi181.stderr:15:55:14 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:55:14.835 INFO:teuthology.orchestra.run.smithi181.stderr:15:55:14 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread 2024-01-19T15:55:14.838 INFO:teuthology.orchestra.run.smithi181.stderr:15:55:14 - INFO - cbt - Pausing for 60s for idle monitoring. 2024-01-19T15:55:14.838 INFO:teuthology.orchestra.run.smithi181.stderr:15:55:14 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:55:14.838 INFO:teuthology.orchestra.run.smithi181.stderr:15:55:14 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:55:14.838 INFO:teuthology.orchestra.run.smithi181.stderr:15:55:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/idle_monitoring/collectl 2024-01-19T15:55:14.840 INFO:teuthology.orchestra.run.smithi181.stderr:15:55:14 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:55:14.840 INFO:teuthology.orchestra.run.smithi181.stderr:15:55:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/idle_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-01-19T15:55:14.842 INFO:teuthology.orchestra.run.smithi181.stderr:15:55:14 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:55:14.842 INFO:teuthology.orchestra.run.smithi181.stderr:15:55:14 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:55:14.842 INFO:teuthology.orchestra.run.smithi181.stderr:15:55:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/idle_monitoring/perf 2024-01-19T15:55:14.845 INFO:teuthology.orchestra.run.smithi181.stderr:15:55:14 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:55:14.845 INFO:teuthology.orchestra.run.smithi181.stderr:15:55:14 - DEBUG - cbt - PerfMonitoring: in local_node 2024-01-19T15:55:14.845 INFO:teuthology.orchestra.run.smithi181.stderr:15:55:14 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-01-19T15:56:14.907 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:14 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:56:14.907 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:14 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:56:14.907 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-01-19T15:56:14.953 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:14 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:56:14.953 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:14 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:56:14.953 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:14 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-01-19T15:56:14.988 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:14 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:56:14.989 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:14 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:56:14.989 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:14 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/* 2024-01-19T15:56:15.007 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:15 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:56:15.007 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:15 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433/$(basename ${i}).smithi181.front.sepia.ceph.com ; done 2024-01-19T15:56:15.011 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:15 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:56:15.011 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:15 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:56:15.011 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/pool_monitoring/collectl 2024-01-19T15:56:15.014 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:15 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:56:15.014 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/pool_monitoring/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-01-19T15:56:15.016 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:15 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:56:15.016 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:15 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:56:15.016 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:15 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/pool_monitoring/perf 2024-01-19T15:56:15.019 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:15 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:56:15.019 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:15 - DEBUG - cbt - PerfMonitoring: in local_node 2024-01-19T15:56:15.019 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:15 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-01-19T15:56:15.019 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:15 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:56:15.019 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:15 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:56:15.019 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:15 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool delete cbt-librbdfio cbt-librbdfio --yes-i-really-really-mean-it 2024-01-19T15:56:15.320 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:15 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:56:15.320 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:15 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:56:15.320 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:15 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool create cbt-librbdfio 128 128 2024-01-19T15:56:16.261 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:16 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:56:16.261 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:16 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:56:16.261 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:16 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool application enable cbt-librbdfio rbd 2024-01-19T15:56:17.263 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:17 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:56:17.263 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:17 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:56:17.263 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:17 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio size 3 2024-01-19T15:56:18.270 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:18 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:56:18.270 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:18 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:56:18.270 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:18 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf osd pool set cbt-librbdfio min_size 2 2024-01-19T15:56:19.271 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:19 - INFO - cbt - Checking Health after pool creation. 2024-01-19T15:56:24.277 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:24 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:56:24.277 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:24 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:56:24.277 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:24 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-01-19T15:56:24.590 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:24 - INFO - cbt - Final Pool Health Check. 2024-01-19T15:56:29.596 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:29 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:56:29.596 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:29 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:56:29.596 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:29 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=/usr/bin/ceph -c /etc/ceph/ceph.conf health 2024-01-19T15:56:29.898 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:29 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:56:29.898 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:29 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:56:29.898 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:29 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=hostname -f 2024-01-19T15:56:29.900 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:29 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:56:29.900 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:29 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:56:29.901 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:29 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi181.front.sepia.ceph.com-0 --size 4096 --pool cbt-librbdfio --order 22 2024-01-19T15:56:31.631 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:31 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:56:31.631 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:31 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:56:31.631 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:31 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=/usr/bin/rbd -c /etc/ceph/ceph.conf create cbt-librbdfio-smithi181.front.sepia.ceph.com-1 --size 4096 --pool cbt-librbdfio --order 22 2024-01-19T15:56:31.731 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:31 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:56:31.732 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:31 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:56:31.732 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:31 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-01-19T15:56:31.749 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:31 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:56:31.749 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:31 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:56:31.749 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:31 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-01-19T15:56:31.778 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:31 - INFO - cbt - Attempting to populating fio files... 2024-01-19T15:56:31.778 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:31 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:56:31.778 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:31 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:56:31.778 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:31 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2024-01-19T15:56:31.780 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:31 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:56:31.780 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:31 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:56:31.780 INFO:teuthology.orchestra.run.smithi181.stderr:15:56:31 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=write --numjobs=1 --bs=4M --size 4096M --name=cbt-librbdfio-`hostname -f`-file-0 --output-format=json,normal > /dev/null 2024-01-19T15:58:59.598 INFO:teuthology.orchestra.run.smithi181.stderr:15:58:59 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:58:59.598 INFO:teuthology.orchestra.run.smithi181.stderr:15:58:59 - INFO - cbt - Skipping existing test in /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433. 2024-01-19T15:58:59.599 INFO:teuthology.orchestra.run.smithi181.stderr:15:58:59 - INFO - cbt - Setting OSD Read Ahead to: 4096 2024-01-19T15:58:59.599 INFO:teuthology.orchestra.run.smithi181.stderr:15:58:59 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:58:59.599 INFO:teuthology.orchestra.run.smithi181.stderr:15:58:59 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:58:59.599 INFO:teuthology.orchestra.run.smithi181.stderr:15:58:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /dev/disk/by-partlabel/osd-device-*data -exec readlink {} \; | cut -d"/" -f 3 | sed "s/[0-9]$//" | xargs -I{} sudo sh -c "echo 4096 > /sys/block/'{}'/queue/read_ahead_kb" 2024-01-19T15:58:59.600 INFO:teuthology.orchestra.run.smithi181.stderr:15:58:59 - DEBUG - cbt - Cleaning existing temporary run directory: /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread 2024-01-19T15:58:59.600 INFO:teuthology.orchestra.run.smithi181.stderr:15:58:59 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:58:59.600 INFO:teuthology.orchestra.run.smithi181.stderr:15:58:59 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:58:59.600 INFO:teuthology.orchestra.run.smithi181.stderr:15:58:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread 2024-01-19T15:58:59.615 INFO:teuthology.orchestra.run.smithi181.stderr:15:58:59 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:58:59.615 INFO:teuthology.orchestra.run.smithi181.stderr:15:58:59 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:58:59.615 INFO:teuthology.orchestra.run.smithi181.stderr:15:58:59 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sync 2024-01-19T15:59:01.032 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:01 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:59:01.032 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo 3 | sudo tee /proc/sys/vm/drop_caches 2024-01-19T15:59:01.822 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:01 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:59:01.822 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:01 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:59:01.823 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:01 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread 2024-01-19T15:59:01.943 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:01 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:59:01.943 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:01 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:59:01.943 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:01 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /usr/bin/ceph -c /etc/ceph/ceph.conf daemon osd.0 config show > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/ceph_settings.out 2024-01-19T15:59:27.928 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:27 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:59:27.928 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:27 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:59:27.928 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:27 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=echo "[`date`] ceph progress" >> /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/pgautoscaler.log;/usr/bin/ceph -c /etc/ceph/ceph.conf progress | tee -a /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/pgautoscaler.log 2024-01-19T15:59:28.269 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:28 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:59:28.270 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:28 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:59:28.270 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/collectl 2024-01-19T15:59:28.272 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:28 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:59:28.272 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=collectl -s+mYZ -i 1:10 -F0 -f /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/collectl --rawdskfilt \"+cciss/c\d+d\d+ |hd[ab] | sd[a-z]+ |dm-\d+ |xvd[a-z] |fio[a-z]+ | vd[a-z]+ |emcpower[a-z]+ |psv\d+ |nvme[0-9]n[0-9]+p[0-9]+ \" 2024-01-19T15:59:28.273 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:28 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:59:28.274 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:28 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:59:28.274 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=mkdir -p -m0755 -- /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/perf 2024-01-19T15:59:28.277 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:28 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:59:28.277 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:28 - DEBUG - cbt - PerfMonitoring: in local_node 2024-01-19T15:59:28.277 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:28 - DEBUG - cbt - pid_dir: /var/run/ceph 2024-01-19T15:59:28.277 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:28 - INFO - cbt - Running rbd fio randread test. 2024-01-19T15:59:28.277 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:28 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-0 2024-01-19T15:59:28.277 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:28 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:59:28.277 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:28 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:59:28.277 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-0 --invalidate=0 --rw=randread --output-format=json,normal --runtime=60 --numjobs=1 --direct=1 --bs=4194304B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/output.0 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/output.0 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/output.0 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/output.0 2024-01-19T15:59:28.278 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:28 - DEBUG - cbt - Using rbdname cbt-librbdfio-`hostname -f`-1 2024-01-19T15:59:28.278 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:28 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:59:28.278 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:28 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:59:28.278 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:28 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo /home/ubuntu/cephtest/fio/fio --ioengine=rbd --clientname=admin --pool=cbt-librbdfio --rbdname=cbt-librbdfio-`hostname -f`-1 --invalidate=0 --rw=randread --output-format=json,normal --runtime=60 --numjobs=1 --direct=1 --bs=4194304B --iodepth=32 --end_fsync=0 --norandommap --write_iops_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/output.1 --write_bw_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/output.1 --write_lat_log=/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/output.1 --log_avg_msec=100 --name=cbt-librbdfio-`hostname -f`-file-0 > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/output.1 2024-01-19T15:59:45.820 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:45 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:59:45.820 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:45 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:59:45.820 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=pkill -SIGINT -f collectl 2024-01-19T15:59:45.847 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:45 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:59:45.847 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:45 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:59:45.847 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo pkill -SIGINT -f perf\ 2024-01-19T15:59:45.877 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:45 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:59:45.877 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/perf/perf.data 2024-01-19T15:59:45.878 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:45 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:59:45.878 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=sudo chown ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/perf/perf_stat.* 2024-01-19T15:59:45.880 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:45 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:59:45.880 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:45 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:59:45.880 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:45 - DEBUG - cbt - CheckedPopen continue_if_error=True, shell=True args=find /var/run/ceph/ceph-osd*.asok -maxdepth 1 -exec sudo /usr/bin/ceph --admin-daemon {} dump_historic_ops \; > /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/historic_ops.out 2024-01-19T15:59:46.178 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:46 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:59:46.192 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:46 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:59:46.193 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:46 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=sudo chown -R ubuntu.ubuntu /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/* 2024-01-19T15:59:46.208 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:46 - DEBUG - cbt - get_fqdn_local()=smithi181.front.sepia.ceph.com 2024-01-19T15:59:46.208 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:46 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=True args=for i in /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread/* ; do cp -r ${i} /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433/$(basename ${i}).smithi181.front.sepia.ceph.com ; done 2024-01-19T15:59:46.251 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:46 - INFO - cbt - Convert results to json format. 2024-01-19T15:59:46.251 INFO:teuthology.orchestra.run.smithi181.stderr:15:59:46 - DEBUG - cbt - Nodes : ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T15:59:46.252 INFO:teuthology.orchestra.run.smithi181.stdout:cleaning remote dir /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randread 2024-01-19T15:59:46.252 INFO:teuthology.orchestra.run.smithi181.stdout:smithi181.front.sepia.ceph.com: smithi181.front.sepia.ceph.com 2024-01-19T15:59:46.252 INFO:teuthology.orchestra.run.smithi181.stdout: 2024-01-19T15:59:46.252 INFO:teuthology.orchestra.run.smithi181.stdout:['smithi181.front.sepia.ceph.com'] 2024-01-19T15:59:46.275 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2024-01-19T15:59:46.285 DEBUG:teuthology.orchestra.run.smithi181:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2024-01-19T15:59:46.433 DEBUG:teuthology.orchestra.run.smithi181:> rm --one-file-system -rf -- /home/ubuntu/cephtest/fio 2024-01-19T15:59:46.499 INFO:tasks.cbt_performance:Collecting CBT performance data config 2024-01-19T15:59:46.499 INFO:tasks.cbt_performance:reading cbt preferences 2024-01-19T15:59:46.499 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest -name 'json_output.*' 2024-01-19T15:59:46.571 INFO:teuthology.orchestra.run.smithi181.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433/json_output.0.smithi181.front.sepia.ceph.com 2024-01-19T15:59:46.571 INFO:teuthology.orchestra.run.smithi181.stdout:/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433/json_output.1.smithi181.front.sepia.ceph.com 2024-01-19T15:59:46.572 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-01-19T15:59:46.572 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433/json_output.0.smithi181.front.sepia.ceph.com of=/dev/stdout 2024-01-19T15:59:46.589 INFO:tasks.cbt_performance:reading total cpu cycles 2024-01-19T15:59:46.589 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433 -name 'perf_stat.*' 2024-01-19T15:59:46.644 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-01-19T15:59:46.644 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-01-19T15:59:46.644 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2024-01-19T15:59:46.644 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433/json_output.1.smithi181.front.sepia.ceph.com of=/dev/stdout 2024-01-19T15:59:46.700 INFO:tasks.cbt_performance:reading total cpu cycles 2024-01-19T15:59:46.701 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/cbt/results/00000000/id-3eb54433 -name 'perf_stat.*' 2024-01-19T15:59:46.754 INFO:tasks.cbt_performance:cpu_cycles_paths: [''] 2024-01-19T15:59:46.755 INFO:tasks.cbt_performance:total cpu cycles: 0 2024-01-19T15:59:46.765 INFO:tasks.cbt_performance:Data inserted successfully. 2024-01-19T15:59:46.774 INFO:tasks.cbt_performance:Data inserted successfully. 2024-01-19T15:59:46.774 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-19T15:59:46.783 DEBUG:teuthology.orchestra.run.smithi181:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-01-19T15:59:46.901 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:59:46.905+0000 7fb6e06ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:59:46.903 INFO:teuthology.orchestra.run.smithi181.stderr:2024-01-19T15:59:46.912+0000 7fb6e06ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-01-19T15:59:47.098 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-19T15:59:47.098 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-19T15:59:47.098 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-19T15:59:47.098 INFO:teuthology.orchestra.run:waiting for 300 2024-01-19T15:59:50.331 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-01-19T15:59:53.199 INFO:tasks.ceph.osd.0:Stopped 2024-01-19T15:59:53.199 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-19T15:59:53.200 INFO:teuthology.orchestra.run:waiting for 300 2024-01-19T15:59:55.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-01-19T15:59:55.634 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-01-19T15:59:59.301 INFO:tasks.ceph.osd.1:Stopped 2024-01-19T15:59:59.301 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-19T15:59:59.301 INFO:teuthology.orchestra.run:waiting for 300 2024-01-19T16:00:00.836 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-01-19T16:00:00.836 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-01-19T16:00:05.403 INFO:tasks.ceph.osd.2:Stopped 2024-01-19T16:00:05.403 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-19T16:00:05.403 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-19T16:00:05.404 INFO:teuthology.orchestra.run:waiting for 300 2024-01-19T16:00:05.436 INFO:tasks.ceph.mgr.x:Stopped 2024-01-19T16:00:05.436 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-19T16:00:05.437 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-19T16:00:05.437 INFO:teuthology.orchestra.run:waiting for 300 2024-01-19T16:00:05.489 INFO:tasks.ceph.mon.a:Stopped 2024-01-19T16:00:05.489 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-19T16:00:05.490 DEBUG:teuthology.orchestra.run.smithi181:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v 'overall HEALTH' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-01-19T16:00:05.578 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T16:00:05.578 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-01-19T16:00:05.777 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T16:00:05.778 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-01-19T16:00:05.837 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-01-19T16:00:05.863 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T16:00:05.863 DEBUG:teuthology.orchestra.run.smithi181:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-01-19T16:00:05.934 INFO:tasks.ceph:Archiving mon data... 2024-01-19T16:00:05.935 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yingxin-2024-01-19_07:32:18-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-8-distro-default-smithi/7521663/data/mon.a.tgz 2024-01-19T16:00:05.935 DEBUG:teuthology.orchestra.run.smithi181:> mktemp 2024-01-19T16:00:05.961 INFO:teuthology.orchestra.run.smithi181.stdout:/tmp/tmp.Mo8wkh9YJ2 2024-01-19T16:00:05.961 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.Mo8wkh9YJ2 2024-01-19T16:00:06.090 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 0666 /tmp/tmp.Mo8wkh9YJ2 2024-01-19T16:00:06.188 DEBUG:teuthology.orchestra.remote:smithi181:/tmp/tmp.Mo8wkh9YJ2 is 194KB 2024-01-19T16:00:06.242 DEBUG:teuthology.orchestra.run.smithi181:> rm -fr /tmp/tmp.Mo8wkh9YJ2 2024-01-19T16:00:06.256 INFO:tasks.ceph:Cleaning ceph cluster... 2024-01-19T16:00:06.257 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-19T16:00:06.386 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-01-19T16:00:06.386 INFO:tasks.ceph:Archiving crash dumps... 2024-01-19T16:00:06.388 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2024-01-19_07:32:18-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-8-distro-default-smithi/7521663/remote/smithi181/crash 2024-01-19T16:00:06.389 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-19T16:00:06.452 INFO:tasks.ceph:Compressing logs... 2024-01-19T16:00:06.452 DEBUG:teuthology.orchestra.run.smithi181:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-19T16:00:06.528 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30122.log 2024-01-19T16:00:06.528 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33945.log 2024-01-19T16:00:06.528 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30122.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-01-19T16:00:06.541 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34458.log 2024-01-19T16:00:06.541 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29976.log 2024-01-19T16:00:06.541 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34065.log 2024-01-19T16:00:06.541 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34458.log.gz 2024-01-19T16:00:06.541 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30143.log 2024-01-19T16:00:06.541 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29976.log: /var/log/ceph/ceph-client.admin.34065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30102.log 2024-01-19T16:00:06.541 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33854.log 2024-01-19T16:00:06.541 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34082.log 2024-01-19T16:00:06.542 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30102.log: /var/log/ceph/ceph-client.admin.33854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29766.log 2024-01-19T16:00:06.542 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28692.log 2024-01-19T16:00:06.542 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30270.log 2024-01-19T16:00:06.542 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28890.log 2024-01-19T16:00:06.542 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.28692.log: /var/log/ceph/ceph-client.admin.30270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34792.log 2024-01-19T16:00:06.542 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.28890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34113.log 2024-01-19T16:00:06.542 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30228.log 2024-01-19T16:00:06.542 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34792.log.gz 2024-01-19T16:00:06.542 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34113.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-19T16:00:06.542 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29565.log 2024-01-19T16:00:06.542 INFO:teuthology.orchestra.run.smithi181.stderr: 66.7% -- replaced with /var/log/ceph/ceph-client.admin.34065.log.gz 2024-01-19T16:00:06.542 INFO:teuthology.orchestra.run.smithi181.stderr:gzip/var/log/ceph/ceph.audit.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29890.log 2024-01-19T16:00:06.542 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29128.log 2024-01-19T16:00:06.542 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29889.log 2024-01-19T16:00:06.543 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29128.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-01-19T16:00:06.543 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34505.log 2024-01-19T16:00:06.543 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28780.log 2024-01-19T16:00:06.543 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.34505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29888.log 2024-01-19T16:00:06.543 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34505.log.gz 2024-01-19T16:00:06.543 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5/var/log/ceph/ceph-client.admin.28780.log: --verbose -- /var/log/ceph/ceph-client.admin.34018.log 2024-01-19T16:00:06.543 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34506.log 2024-01-19T16:00:06.543 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34041.log 2024-01-19T16:00:06.543 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28930.log 2024-01-19T16:00:06.543 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34506.log.gz 2024-01-19T16:00:06.543 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30122.log.gz 2024-01-19T16:00:06.543 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33995.log 2024-01-19T16:00:06.543 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30143.log.gz 2024-01-19T16:00:06.543 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.28930.log: 65.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.34082.log.gz 2024-01-19T16:00:06.544 INFO:teuthology.orchestra.run.smithi181.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.28970.log 2024-01-19T16:00:06.544 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30438.log 2024-01-19T16:00:06.544 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33995.log: /var/log/ceph/ceph-client.admin.28970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30354.log 2024-01-19T16:00:06.544 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34113.log.gz 2024-01-19T16:00:06.544 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30438.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-01-19T16:00:06.544 INFO:teuthology.orchestra.run.smithi181.stderr:gzip/var/log/ceph/ceph-client.admin.30354.log: -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27321.log 2024-01-19T16:00:06.544 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30228.log.gz 2024-01-19T16:00:06.544 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30016.log 2024-01-19T16:00:06.544 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30270.log.gz 2024-01-19T16:00:06.544 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph.tmp-client.admin.27321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29158.log 2024-01-19T16:00:06.544 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34114.log 2024-01-19T16:00:06.544 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29158.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34018.log.gz 2024-01-19T16:00:06.544 INFO:teuthology.orchestra.run.smithi181.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-01-19T16:00:06.544 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.34114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30015.log 2024-01-19T16:00:06.545 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.34041.log.gz 2024-01-19T16:00:06.545 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30438.log.gz 2024-01-19T16:00:06.545 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33918.log 2024-01-19T16:00:06.545 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30015.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-19T16:00:06.545 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30354.log.gz 2024-01-19T16:00:06.545 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29808.log 2024-01-19T16:00:06.545 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-01-19T16:00:06.545 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29663.log 2024-01-19T16:00:06.545 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29621.log 2024-01-19T16:00:06.545 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.34114.log.gz 2024-01-19T16:00:06.545 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30248.log 2024-01-19T16:00:06.545 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29724.log 2024-01-19T16:00:06.545 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.30248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33970.log 2024-01-19T16:00:06.545 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.29724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30396.log 2024-01-19T16:00:06.547 INFO:teuthology.orchestra.run.smithi181.stderr:/var/log/ceph/ceph-client.admin.33970.log: /var/log/ceph/ceph-client.admin.30396.log: 66.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27321.log.gz 2024-01-19T16:00:06.548 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30248.log.gz 2024-01-19T16:00:06.552 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29565.log.gz 2024-01-19T16:00:06.552 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29621.log.gz 2024-01-19T16:00:06.552 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29663.log.gz 2024-01-19T16:00:06.552 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29724.log.gz 2024-01-19T16:00:06.552 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29766.log.gz 2024-01-19T16:00:06.552 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29808.log.gz 2024-01-19T16:00:06.552 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29890.log.gz 2024-01-19T16:00:06.552 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29888.log.gz 2024-01-19T16:00:06.552 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29889.log.gz 2024-01-19T16:00:06.552 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29976.log.gz 2024-01-19T16:00:06.552 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.30016.log.gz 2024-01-19T16:00:06.552 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30015.log.gz 2024-01-19T16:00:06.553 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30102.log.gz 2024-01-19T16:00:06.553 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.30396.log.gz 2024-01-19T16:00:06.566 INFO:teuthology.orchestra.run.smithi181.stderr: 87.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-19T16:00:06.566 INFO:teuthology.orchestra.run.smithi181.stderr: 50.4% 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28970.log.gz -- replaced with /var/log/ceph/ceph-client.admin.29128.log.gz 2024-01-19T16:00:06.566 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-01-19T16:00:06.566 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29158.log.gz 2024-01-19T16:00:06.566 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28692.log.gz 91.1% 2024-01-19T16:00:06.566 INFO:teuthology.orchestra.run.smithi181.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28930.log.gz -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-01-19T16:00:06.566 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-01-19T16:00:06.566 INFO:teuthology.orchestra.run.smithi181.stderr: 50.0% 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28890.log.gz -- replaced with /var/log/ceph/ceph-client.admin.28780.log.gz 2024-01-19T16:00:06.567 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-01-19T16:00:06.581 INFO:teuthology.orchestra.run.smithi181.stderr: 86.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-19T16:00:06.588 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33854.log.gz 2024-01-19T16:00:06.588 INFO:teuthology.orchestra.run.smithi181.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.33918.log.gz 2024-01-19T16:00:06.629 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33945.log.gz 2024-01-19T16:00:06.630 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33970.log.gz 2024-01-19T16:00:06.630 INFO:teuthology.orchestra.run.smithi181.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.33995.log.gz 2024-01-19T16:00:06.635 INFO:teuthology.orchestra.run.smithi181.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-01-19T16:00:24.013 INFO:teuthology.orchestra.run.smithi181.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-01-19T16:00:24.061 INFO:teuthology.orchestra.run.smithi181.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-01-19T16:00:24.109 INFO:teuthology.orchestra.run.smithi181.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-01-19T16:00:24.111 INFO:teuthology.orchestra.run.smithi181.stderr: 2024-01-19T16:00:24.111 INFO:teuthology.orchestra.run.smithi181.stderr:real 0m17.605s 2024-01-19T16:00:24.111 INFO:teuthology.orchestra.run.smithi181.stderr:user 0m9.679s 2024-01-19T16:00:24.111 INFO:teuthology.orchestra.run.smithi181.stderr:sys 0m0.527s 2024-01-19T16:00:24.112 INFO:tasks.ceph:Archiving logs... 2024-01-19T16:00:24.112 DEBUG:teuthology.misc:Transferring archived files from smithi181:/var/log/ceph to /home/teuthworker/archive/yingxin-2024-01-19_07:32:18-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-8-distro-default-smithi/7521663/remote/smithi181/log 2024-01-19T16:00:24.113 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-19T16:00:25.703 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-19T16:00:25.713 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-19T16:00:25.713 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-19T16:00:25.768 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-19T16:00:25.777 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-19T16:00:25.778 DEBUG:teuthology.orchestra.run.smithi181:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-19T16:00:25.798 INFO:teuthology.orchestra.run.smithi181.stderr:bash: line 1: ntpq: command not found 2024-01-19T16:00:25.881 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-19T16:00:25.881 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2024-01-19T16:00:25.881 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 4 -417us[ -417us] +/- 75ms 2024-01-19T16:00:25.881 INFO:teuthology.orchestra.run.smithi181.stdout:^* hv02.front.sepia.ceph.com 3 6 377 5 -1527us[-1392us] +/- 52ms 2024-01-19T16:00:25.881 INFO:teuthology.orchestra.run.smithi181.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 +2384us[+2384us] +/- 71ms 2024-01-19T16:00:25.881 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-19T16:00:25.900 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-19T16:00:25.910 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-19T16:00:25.911 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-19T16:00:25.922 DEBUG:teuthology.orchestra.run.smithi181:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-19T16:00:26.068 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-19T16:00:26.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-19T16:00:26.098 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-19T16:00:26.108 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-19T16:00:26.141 INFO:teuthology.task.internal:Duration was 833.391161 seconds 2024-01-19T16:00:26.141 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-19T16:00:26.153 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-19T16:00:26.154 DEBUG:teuthology.orchestra.run.smithi181:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-19T16:00:26.306 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-19T16:00:26.853 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-19T16:00:26.853 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi181.front.sepia.ceph.com 2024-01-19T16:00:26.853 DEBUG:teuthology.orchestra.run.smithi181:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-19T16:00:26.873 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-19T16:00:26.873 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-19T16:00:26.942 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-19T16:00:26.942 DEBUG:teuthology.orchestra.run.smithi181:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-19T16:00:27.085 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-19T16:00:27.095 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-19T16:00:27.095 DEBUG:teuthology.orchestra.run.smithi181:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-19T16:00:27.126 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-19T16:00:27.137 DEBUG:teuthology.orchestra.run.smithi181:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-19T16:00:27.207 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = core 2024-01-19T16:00:27.230 DEBUG:teuthology.orchestra.run.smithi181:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-19T16:00:27.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-19T16:00:27.284 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-19T16:00:27.294 INFO:teuthology.task.internal:Transferring archived files... 2024-01-19T16:00:27.295 DEBUG:teuthology.misc:Transferring archived files from smithi181:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yingxin-2024-01-19_07:32:18-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-8-distro-default-smithi/7521663/remote/smithi181 2024-01-19T16:00:27.295 DEBUG:teuthology.orchestra.run.smithi181:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-19T16:00:27.391 INFO:teuthology.task.internal:Removing archive directory... 2024-01-19T16:00:27.391 DEBUG:teuthology.orchestra.run.smithi181:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-19T16:00:27.447 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-19T16:00:27.460 INFO:teuthology.task.internal:Not uploading archives. 2024-01-19T16:00:27.460 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-19T16:00:27.479 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-19T16:00:27.479 DEBUG:teuthology.orchestra.run.smithi181:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-19T16:00:27.501 INFO:teuthology.orchestra.run.smithi181.stdout: 265568 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 19 16:00 /home/ubuntu/cephtest 2024-01-19T16:00:27.502 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-19T16:00:27.513 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-19T16:00:27.536 INFO:teuthology.run:Summary data: cbt_perf_url: http://mira118.front.sepia.ceph.com:3000/d/2Jx1MmfVk/fio?orgId=1&var-branch_name=wip-yingxin-crimson-save-conn-foreign-copy-8&var-job_id_selected=7521663 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read} duration: 833.391161441803 flavor: crimson owner: scheduled_yingxin@teuthology success: true 2024-01-19T16:00:27.537 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-19T16:00:27.599 INFO:teuthology.run:pass