2024-04-22T01:49:43.593 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-22T01:49:43.593 DEBUG:teuthology.run:Teuthology command: teuthology --name yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi --verbose --owner scheduled_yingxin@teuthology --description crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} --archive /home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668110 -- /home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668110/orig.config.yaml 2024-04-22T01:49:43.640 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T01:49:43.746 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668110 branch: ci-yingxin-crimson-fix-tri-mutex-3 description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} email: yingxin.cheng@intel.com first_in_suite: false job_id: '7668110' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: ci-yingxin-crimson-fix-tri-mutex-3 ceph: conf: global: auth client required: none auth cluster required: none auth service required: none auth supported: none debug asok: 0/0 debug auth: 0/0 debug bdev: 0/0 debug bluefs: 0/0 debug bluestore: 0/0 debug buffer: 0/0 debug civetweb: 0/0 debug client: 0/0 debug compressor: 0/0 debug context: 0/0 debug crush: 0/0 debug crypto: 0/0 debug dpdk: 0/0 debug eventtrace: 0/0 debug filer: 0/0 debug filestore: 0/0 debug finisher: 0/0 debug fuse: 0/0 debug heartbeatmap: 0/0 debug javaclient: 0/0 debug journal: 0/0 debug journaler: 0/0 debug kstore: 0/0 debug leveldb: 0/0 debug lockdep: 0/0 debug mds: 0/0 debug mds balancer: 0/0 debug mds locker: 0/0 debug mds log: 0/0 debug mds log expire: 0/0 debug mds migrator: 0/0 debug memdb: 0/0 debug mgr: 0/0 debug mgrc: 0/0 debug mon: 0/0 debug monc: 0/0 debug ms: 0/0 debug objclass: 0/0 debug objectcacher: 0/0 debug objecter: 0/0 debug optracker: 0/0 debug osd: 0/0 debug paxos: 0/0 debug perfcounter: 0/0 debug rados: 0/0 debug rbd: 0/0 debug rbd mirror: 0/0 debug rbd replay: 0/0 debug refs: 0/0 debug rgw: 0/0 debug rgw lifecycle: 0/0 debug rgw notification: 0/0 debug rgw sync: 0/0 debug rocksdb: 0/0 debug striper: 0/0 debug throttle: 0/0 debug timer: 0/0 debug tp: 0/0 enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 0/0 debug ms: 0/0 debug paxos: 0/0 osd pool default crimson: true osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore rocksdb cf: false crimson alien thread cpu cores: 6-7 crimson osd obc lru size: 10 debug bluefs: 20 debug bluestore: 20 debug filestore: 0/0 debug journal: 0/0 debug ms: 0/0 debug osd: 0/0 debug rocksdb: 10 err to stderr: true log flush on exit: true log to file: false log to stderr: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true osd.0: crimson seastar cpu cores: 0-2 osd.1: crimson seastar cpu cores: 3-5 osd.2: crimson seastar cpu cores: 0-2 osd.3: crimson seastar cpu cores: 3-5 flavor: crimson fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_ - \(OSD_ - \(OBJECT_ - overall HEALTH - \(POOL_APP_NOT_ENABLED\) sha1: 676947e1f73a9d7f900df5a6854c283c4eef24a9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: crimson sha1: 676947e1f73a9d7f900df5a6854c283c4eef24a9 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 workunit: branch: ci-yingxin-crimson-fix-tri-mutex-3 env: CRIMSON_COMPAT: '1' sha1: 676947e1f73a9d7f900df5a6854c283c4eef24a9 owner: scheduled_yingxin@teuthology priority: 60 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 676947e1f73a9d7f900df5a6854c283c4eef24a9 sleep_before_teardown: 0 suite: crimson-rados suite_branch: ci-yingxin-crimson-fix-tri-mutex-3 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 676947e1f73a9d7f900df5a6854c283c4eef24a9 targets: smithi152.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK7vEJZAHyaYpnTOgKeOB12u7qU8K1guXN09pp+HLdP6rPm6XeYpmk9p/CF/C2mX0lNi+GqDfPWdLPJ+2dbhBvU= tasks: - install: null - ceph: conf: osd: debug monc: 20 flavor: crimson wait-for-scrub: false - cbt: benchmarks: librbdfio: iodepth: - 32 log_avg_msec: 100 mode: - randrw norandommap: true op_size: - 4194304 osd_ra: - 4096 pool_profile: rbd procs_per_volume: - 1 time: 60 vol_size: 4096 volumes_per_client: - 2 cluster: iterations: 1 osds_per_node: 3 pool_profiles: rbd: pg_size: 128 pgp_size: 128 replication: 3 user: ubuntu monitoring_profiles: perf: args: stat -p {pid} -o {perf_dir}/perf_stat.{pid} nodes: - osds perf_cmd: perf pid_glob: crimson-osd.*.pid teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-22_01:37:40 tube: smithi user: yingxin verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-22T01:49:43.747 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa; will attempt to use it 2024-04-22T01:49:43.747 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks 2024-04-22T01:49:43.747 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-22T01:49:43.749 INFO:teuthology.task.internal:Checking packages... 2024-04-22T01:49:43.773 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'crimson' and ceph hash '676947e1f73a9d7f900df5a6854c283c4eef24a9' 2024-04-22T01:49:43.773 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T01:49:43.774 INFO:teuthology.packaging:ref: None 2024-04-22T01:49:43.774 INFO:teuthology.packaging:tag: None 2024-04-22T01:49:43.774 INFO:teuthology.packaging:branch: ci-yingxin-crimson-fix-tri-mutex-3 2024-04-22T01:49:43.774 INFO:teuthology.packaging:sha1: 676947e1f73a9d7f900df5a6854c283c4eef24a9 2024-04-22T01:49:43.774 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&ref=ci-yingxin-crimson-fix-tri-mutex-3 2024-04-22T01:49:43.921 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3159.g676947e1 2024-04-22T01:49:43.924 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-22T01:49:43.930 INFO:teuthology.task.internal:no buildpackages task found 2024-04-22T01:49:43.931 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-22T01:49:43.944 INFO:teuthology.task.internal:Saving configuration 2024-04-22T01:49:43.961 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-22T01:49:43.967 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T01:49:43.988 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668110', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 01:44:23.175631', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK7vEJZAHyaYpnTOgKeOB12u7qU8K1guXN09pp+HLdP6rPm6XeYpmk9p/CF/C2mX0lNi+GqDfPWdLPJ+2dbhBvU='} 2024-04-22T01:49:43.988 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-22T01:49:43.995 INFO:teuthology.task.internal:roles: ubuntu@smithi152.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-04-22T01:49:43.995 INFO:teuthology.run_tasks:Running task console_log... 2024-04-22T01:49:44.032 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0359472e50>, signals=[15]) 2024-04-22T01:49:44.032 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-22T01:49:44.039 INFO:teuthology.task.internal:Opening connections... 2024-04-22T01:49:44.039 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi152.front.sepia.ceph.com 2024-04-22T01:49:44.041 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T01:49:44.119 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-22T01:49:44.125 DEBUG:teuthology.orchestra.run.smithi152:> uname -m 2024-04-22T01:49:44.211 INFO:teuthology.orchestra.run.smithi152.stdout:x86_64 2024-04-22T01:49:44.212 DEBUG:teuthology.orchestra.run.smithi152:> cat /etc/os-release 2024-04-22T01:49:44.267 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="CentOS Stream" 2024-04-22T01:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="9" 2024-04-22T01:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout:ID="centos" 2024-04-22T01:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE="rhel fedora" 2024-04-22T01:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="9" 2024-04-22T01:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout:PLATFORM_ID="platform:el9" 2024-04-22T01:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T01:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout:ANSI_COLOR="0;31" 2024-04-22T01:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout:LOGO="fedora-logo-icon" 2024-04-22T01:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T01:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://centos.org/" 2024-04-22T01:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T01:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T01:49:44.268 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T01:49:44.269 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2024-04-22T01:49:44.292 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-22T01:49:44.301 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-22T01:49:44.308 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-22T01:49:44.308 DEBUG:teuthology.orchestra.run.smithi152:> test '!' -e /home/ubuntu/cephtest 2024-04-22T01:49:44.324 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-22T01:49:44.334 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-22T01:49:44.334 DEBUG:teuthology.orchestra.run.smithi152:> test -z $(ls -A /var/lib/ceph) 2024-04-22T01:49:44.396 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-22T01:49:44.440 INFO:teuthology.run_tasks:Running task kernel... 2024-04-22T01:49:44.462 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-22T01:49:44.463 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-22T01:49:44.463 DEBUG:teuthology.orchestra.run.smithi152:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T01:49:44.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:49:44.479 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2024-04-22T01:49:44.535 INFO:teuthology.orchestra.run.smithi152.stdout:5.14.0-437.el9.x86_64 2024-04-22T01:49:44.535 INFO:teuthology.task.kernel:Running kernel on smithi152: 5.14.0-437.el9.x86_64 2024-04-22T01:49:44.536 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum install -y kernel 2024-04-22T01:49:47.077 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - BaseOS 8.6 MB/s | 8.0 MB 00:00 2024-04-22T01:49:48.785 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - AppStream 30 MB/s | 19 MB 00:00 2024-04-22T01:49:54.002 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.2 MB 00:01 2024-04-22T01:49:56.369 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-04-22T01:49:58.412 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-22T01:50:03.512 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-04-22T01:50:05.419 INFO:teuthology.orchestra.run.smithi152.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T01:50:05.419 INFO:teuthology.orchestra.run.smithi152.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T01:50:05.508 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:50:05.510 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-04-22T01:50:05.510 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:50:05.613 DEBUG:teuthology.orchestra.run.smithi152:> echo no | sudo yum reinstall kernel || true 2024-04-22T01:50:06.114 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 01:50:03 AM UTC. 2024-04-22T01:50:06.240 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:50:06.250 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:50:06.250 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-04-22T01:50:06.250 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:50:06.250 INFO:teuthology.orchestra.run.smithi152.stdout:Reinstalling: 2024-04-22T01:50:06.251 INFO:teuthology.orchestra.run.smithi152.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T01:50:06.251 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:50:06.251 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-04-22T01:50:06.251 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:50:06.251 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:50:06.251 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 255 k 2024-04-22T01:50:06.251 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 0 2024-04-22T01:50:06.251 INFO:teuthology.orchestra.run.smithi152.stderr:Operation aborted. 2024-04-22T01:50:06.315 INFO:teuthology.orchestra.run.smithi152.stdout:Is this ok [y/N]: 2024-04-22T01:50:06.316 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum reinstall -y kernel || true 2024-04-22T01:50:06.812 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 01:50:03 AM UTC. 2024-04-22T01:50:06.929 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:50:06.930 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:50:06.930 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-04-22T01:50:06.930 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:50:06.930 INFO:teuthology.orchestra.run.smithi152.stdout:Reinstalling: 2024-04-22T01:50:06.930 INFO:teuthology.orchestra.run.smithi152.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T01:50:06.930 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:50:06.930 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-04-22T01:50:06.930 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:50:06.930 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:50:06.931 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 255 k 2024-04-22T01:50:06.931 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 0 2024-04-22T01:50:06.931 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-04-22T01:50:07.631 INFO:teuthology.orchestra.run.smithi152.stdout:kernel-5.14.0-437.el9.x86_64.rpm 472 kB/s | 255 kB 00:00 2024-04-22T01:50:07.631 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:50:07.632 INFO:teuthology.orchestra.run.smithi152.stdout:Total 363 kB/s | 255 kB 00:00 2024-04-22T01:50:07.651 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-04-22T01:50:07.844 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-04-22T01:50:07.844 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-04-22T01:50:07.849 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-04-22T01:50:07.849 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-04-22T01:50:07.997 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-04-22T01:50:08.066 INFO:teuthology.orchestra.run.smithi152.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T01:50:08.066 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T01:50:08.819 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T01:50:08.819 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T01:50:09.563 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T01:50:09.563 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:50:09.564 INFO:teuthology.orchestra.run.smithi152.stdout:Reinstalled: 2024-04-22T01:50:09.564 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T01:50:09.564 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:50:09.564 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:50:09.719 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T01:50:09.766 INFO:teuthology.orchestra.run.smithi152.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T01:50:09.767 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T01:50:09.767 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T01:50:09.767 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-22T01:50:09.767 DEBUG:teuthology.orchestra.run.smithi152:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T01:50:09.801 INFO:teuthology.orchestra.run.smithi152.stdout:ttyS1 2024-04-22T01:50:09.826 DEBUG:teuthology.parallel:result is None 2024-04-22T01:50:09.826 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-22T01:50:09.833 INFO:teuthology.task.internal:Creating test directory... 2024-04-22T01:50:09.833 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T01:50:09.881 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-22T01:50:09.889 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-22T01:50:09.896 INFO:teuthology.task.internal:Creating archive directory... 2024-04-22T01:50:09.896 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T01:50:09.946 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-22T01:50:09.954 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-22T01:50:09.954 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T01:50:10.069 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T01:50:10.078 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T01:50:10.080 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-22T01:50:10.089 INFO:teuthology.task.internal:Configuring sudo... 2024-04-22T01:50:10.089 DEBUG:teuthology.orchestra.run.smithi152:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T01:50:10.113 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-22T01:50:10.136 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-22T01:50:10.136 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T01:50:10.168 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T01:50:10.244 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T01:50:10.307 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T01:50:10.383 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T01:50:10.428 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-22T01:50:10.428 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T01:50:10.492 DEBUG:teuthology.orchestra.run.smithi152:> sudo service rsyslog restart 2024-04-22T01:50:10.576 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T01:50:10.742 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-22T01:50:10.750 INFO:teuthology.task.internal:Starting timer... 2024-04-22T01:50:10.750 INFO:teuthology.run_tasks:Running task pcp... 2024-04-22T01:50:10.761 INFO:teuthology.run_tasks:Running task selinux... 2024-04-22T01:50:10.773 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-22T01:50:10.773 DEBUG:teuthology.orchestra.run.smithi152:> sudo service auditd rotate 2024-04-22T01:50:10.822 INFO:teuthology.orchestra.run.smithi152.stdout:Rotating logs: 2024-04-22T01:50:10.824 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-22T01:50:10.824 DEBUG:teuthology.orchestra.run.smithi152:> /usr/sbin/getenforce 2024-04-22T01:50:10.843 INFO:teuthology.orchestra.run.smithi152.stdout:Permissive 2024-04-22T01:50:10.843 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi152.front.sepia.ceph.com': 'permissive'} 2024-04-22T01:50:10.844 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T01:50:10.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:50:10.908 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-22T01:50:10.909 DEBUG:teuthology.orchestra.run.smithi152:> sudo /usr/sbin/setenforce permissive 2024-04-22T01:50:10.978 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-22T01:50:10.990 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-22T01:50:11.008 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-22T01:50:11.009 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi152.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-22T01:53:36.640 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi152.front.sepia.ceph.com')] 2024-04-22T01:53:36.641 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-04-22T01:53:36.643 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T01:53:36.733 DEBUG:teuthology.orchestra.run.smithi152:> true 2024-04-22T01:53:36.809 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2024-04-22T01:53:36.810 INFO:teuthology.run_tasks:Running task clock... 2024-04-22T01:53:36.820 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-22T01:53:36.820 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T01:53:36.821 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T01:53:36.877 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T01:53:36.890 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T01:53:36.913 INFO:teuthology.orchestra.run.smithi152.stderr:sudo: ntpd: command not found 2024-04-22T01:53:36.923 INFO:teuthology.orchestra.run.smithi152.stdout:506 Cannot talk to daemon 2024-04-22T01:53:36.936 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T01:53:36.948 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T01:53:37.000 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-04-22T01:53:37.003 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T01:53:37.003 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-04-22T01:53:37.004 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T01:53:37.004 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T01:53:37.004 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T01:53:37.004 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T01:53:37.005 INFO:teuthology.run_tasks:Running task install... 2024-04-22T01:53:37.013 DEBUG:teuthology.task.install:project ceph 2024-04-22T01:53:37.013 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'crimson', 'sha1': '676947e1f73a9d7f900df5a6854c283c4eef24a9'}} 2024-04-22T01:53:37.014 DEBUG:teuthology.task.install:config {'flavor': 'crimson', 'sha1': '676947e1f73a9d7f900df5a6854c283c4eef24a9'} 2024-04-22T01:53:37.014 INFO:teuthology.task.install:Using flavor: crimson 2024-04-22T01:53:37.019 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-22T01:53:37.019 INFO:teuthology.task.install:extra packages: [] 2024-04-22T01:53:37.020 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'crimson', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '676947e1f73a9d7f900df5a6854c283c4eef24a9', 'tag': None, 'wait_for_package': False} 2024-04-22T01:53:37.020 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=676947e1f73a9d7f900df5a6854c283c4eef24a9 2024-04-22T01:53:37.242 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-3/676947e1f73a9d7f900df5a6854c283c4eef24a9/centos/9/flavors/crimson/ 2024-04-22T01:53:37.243 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3159.g676947e1 2024-04-22T01:53:37.398 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-3/676947e1f73a9d7f900df5a6854c283c4eef24a9/centos/9/flavors/crimson/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-3/676947e1f73a9d7f900df5a6854c283c4eef24a9/centos/9/flavors/crimson/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/ci-yingxin-crimson-fix-tri-mutex-3/676947e1f73a9d7f900df5a6854c283c4eef24a9/centos/9/flavors/crimson/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-22T01:53:37.398 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-22T01:53:37.398 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-22T01:53:37.423 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-04-22T01:53:37.423 DEBUG:teuthology.orchestra.run.smithi152:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/676947e1f73a9d7f900df5a6854c283c4eef24a9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-22T01:53:37.488 DEBUG:teuthology.orchestra.run.smithi152:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-22T01:53:37.561 DEBUG:teuthology.orchestra.run.smithi152:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-22T01:53:37.624 INFO:teuthology.orchestra.run.smithi152.stdout:check_obsoletes = 1 2024-04-22T01:53:37.626 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum clean all 2024-04-22T01:53:37.852 INFO:teuthology.orchestra.run.smithi152.stdout:48 files removed 2024-04-22T01:53:37.880 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-radosgw 2024-04-22T01:53:38.552 INFO:teuthology.orchestra.run.smithi152.stdout:ceph packages for x86_64 211 kB/s | 87 kB 00:00 2024-04-22T01:53:38.936 INFO:teuthology.orchestra.run.smithi152.stdout:ceph noarch packages 34 kB/s | 12 kB 00:00 2024-04-22T01:53:39.328 INFO:teuthology.orchestra.run.smithi152.stdout:ceph source packages 5.7 kB/s | 2.0 kB 00:00 2024-04-22T01:53:48.487 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - BaseOS 903 kB/s | 8.0 MB 00:09 2024-04-22T01:53:50.704 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-04-22T01:53:55.230 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - CRB 6.8 MB/s | 6.2 MB 00:00 2024-04-22T01:53:56.631 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 9 - Extras packages 65 kB/s | 16 kB 00:00 2024-04-22T01:53:58.310 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-22T01:54:03.241 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-04-22T01:54:05.425 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:54:05.427 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:54:05.427 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-04-22T01:54:05.427 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:54:05.427 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-04-22T01:54:05.427 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw x86_64 2:19.0.0-3159.g676947e1.el9 ceph 10 M 2024-04-22T01:54:05.427 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrading: 2024-04-22T01:54:05.427 INFO:teuthology.orchestra.run.smithi152.stdout: librados2 x86_64 2:19.0.0-3159.g676947e1.el9 ceph 3.4 M 2024-04-22T01:54:05.427 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1 x86_64 2:19.0.0-3159.g676947e1.el9 ceph 3.7 M 2024-04-22T01:54:05.427 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-04-22T01:54:05.427 INFO:teuthology.orchestra.run.smithi152.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-22T01:54:05.428 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base x86_64 2:19.0.0-3159.g676947e1.el9 ceph 5.9 M 2024-04-22T01:54:05.428 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common x86_64 2:19.0.0-3159.g676947e1.el9 ceph 19 M 2024-04-22T01:54:05.428 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux x86_64 2:19.0.0-3159.g676947e1.el9 ceph 25 k 2024-04-22T01:54:05.428 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-22T01:54:05.428 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-22T01:54:05.428 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-22T01:54:05.428 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2 x86_64 2:19.0.0-3159.g676947e1.el9 ceph 796 k 2024-04-22T01:54:05.428 INFO:teuthology.orchestra.run.smithi152.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-22T01:54:05.428 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-22T01:54:05.428 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-22T01:54:05.428 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1 x86_64 2:19.0.0-3159.g676947e1.el9 ceph 549 k 2024-04-22T01:54:05.428 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-22T01:54:05.428 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2 x86_64 2:19.0.0-3159.g676947e1.el9 ceph 5.3 M 2024-04-22T01:54:05.429 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-22T01:54:05.429 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-22T01:54:05.429 INFO:teuthology.orchestra.run.smithi152.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-22T01:54:05.429 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse x86_64 2:19.0.0-3159.g676947e1.el9 ceph 45 k 2024-04-22T01:54:05.429 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common x86_64 2:19.0.0-3159.g676947e1.el9 ceph 135 k 2024-04-22T01:54:05.429 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs x86_64 2:19.0.0-3159.g676947e1.el9 ceph 160 k 2024-04-22T01:54:05.429 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-22T01:54:05.429 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados x86_64 2:19.0.0-3159.g676947e1.el9 ceph 315 k 2024-04-22T01:54:05.429 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd x86_64 2:19.0.0-3159.g676947e1.el9 ceph 293 k 2024-04-22T01:54:05.429 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw x86_64 2:19.0.0-3159.g676947e1.el9 ceph 99 k 2024-04-22T01:54:05.429 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-22T01:54:05.429 INFO:teuthology.orchestra.run.smithi152.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-22T01:54:05.429 INFO:teuthology.orchestra.run.smithi152.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-22T01:54:05.429 INFO:teuthology.orchestra.run.smithi152.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-22T01:54:05.430 INFO:teuthology.orchestra.run.smithi152.stdout:Installing weak dependencies: 2024-04-22T01:54:05.430 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-22T01:54:05.430 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:54:05.430 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-04-22T01:54:05.430 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:54:05.430 INFO:teuthology.orchestra.run.smithi152.stdout:Install 30 Packages 2024-04-22T01:54:05.430 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrade 2 Packages 2024-04-22T01:54:05.430 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:54:05.430 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 59 M 2024-04-22T01:54:05.431 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-04-22T01:54:07.901 INFO:teuthology.orchestra.run.smithi152.stdout:(1/32): ceph-radosgw-19.0.0-3159.g676947e1.el9. 4.5 MB/s | 10 MB 00:02 2024-04-22T01:54:07.959 INFO:teuthology.orchestra.run.smithi152.stdout:(2/32): ceph-selinux-19.0.0-3159.g676947e1.el9. 432 kB/s | 25 kB 00:00 2024-04-22T01:54:08.168 INFO:teuthology.orchestra.run.smithi152.stdout:(3/32): libcephfs2-19.0.0-3159.g676947e1.el9.x8 3.7 MB/s | 796 kB 00:00 2024-04-22T01:54:08.293 INFO:teuthology.orchestra.run.smithi152.stdout:(4/32): libradosstriper1-19.0.0-3159.g676947e1. 4.3 MB/s | 549 kB 00:00 2024-04-22T01:54:08.702 INFO:teuthology.orchestra.run.smithi152.stdout:(5/32): librgw2-19.0.0-3159.g676947e1.el9.x86_6 13 MB/s | 5.3 MB 00:00 2024-04-22T01:54:08.760 INFO:teuthology.orchestra.run.smithi152.stdout:(6/32): python3-ceph-argparse-19.0.0-3159.g6769 776 kB/s | 45 kB 00:00 2024-04-22T01:54:08.819 INFO:teuthology.orchestra.run.smithi152.stdout:(7/32): python3-ceph-common-19.0.0-3159.g676947 2.2 MB/s | 135 kB 00:00 2024-04-22T01:54:08.886 INFO:teuthology.orchestra.run.smithi152.stdout:(8/32): python3-cephfs-19.0.0-3159.g676947e1.el 2.3 MB/s | 160 kB 00:00 2024-04-22T01:54:08.961 INFO:teuthology.orchestra.run.smithi152.stdout:(9/32): python3-rados-19.0.0-3159.g676947e1.el9 4.1 MB/s | 315 kB 00:00 2024-04-22T01:54:09.028 INFO:teuthology.orchestra.run.smithi152.stdout:(10/32): python3-rbd-19.0.0-3159.g676947e1.el9. 4.3 MB/s | 293 kB 00:00 2024-04-22T01:54:09.087 INFO:teuthology.orchestra.run.smithi152.stdout:(11/32): python3-rgw-19.0.0-3159.g676947e1.el9. 1.7 MB/s | 99 kB 00:00 2024-04-22T01:54:09.512 INFO:teuthology.orchestra.run.smithi152.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 252 kB/s | 107 kB 00:00 2024-04-22T01:54:09.779 INFO:teuthology.orchestra.run.smithi152.stdout:(13/32): ceph-common-19.0.0-3159.g676947e1.el9. 4.7 MB/s | 19 MB 00:04 2024-04-22T01:54:09.863 INFO:teuthology.orchestra.run.smithi152.stdout:(14/32): ceph-base-19.0.0-3159.g676947e1.el9.x8 1.4 MB/s | 5.9 MB 00:04 2024-04-22T01:54:09.888 INFO:teuthology.orchestra.run.smithi152.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 426 kB/s | 160 kB 00:00 2024-04-22T01:54:10.080 INFO:teuthology.orchestra.run.smithi152.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.5 MB/s | 292 kB 00:00 2024-04-22T01:54:10.121 INFO:teuthology.orchestra.run.smithi152.stdout:(17/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 132 kB/s | 45 kB 00:00 2024-04-22T01:54:10.172 INFO:teuthology.orchestra.run.smithi152.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 460 kB/s | 42 kB 00:00 2024-04-22T01:54:10.289 INFO:teuthology.orchestra.run.smithi152.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 307 kB/s | 36 kB 00:00 2024-04-22T01:54:10.389 INFO:teuthology.orchestra.run.smithi152.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 826 kB/s | 221 kB 00:00 2024-04-22T01:54:10.415 INFO:teuthology.orchestra.run.smithi152.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 379 kB/s | 47 kB 00:00 2024-04-22T01:54:10.548 INFO:teuthology.orchestra.run.smithi152.stdout:(22/32): librdkafka-1.6.1-102.el9.x86_64.rpm 967 kB/s | 662 kB 00:00 2024-04-22T01:54:10.573 INFO:teuthology.orchestra.run.smithi152.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-04-22T01:54:10.615 INFO:teuthology.orchestra.run.smithi152.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-04-22T01:54:10.641 INFO:teuthology.orchestra.run.smithi152.stdout:(25/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 281 kB/s | 26 kB 00:00 2024-04-22T01:54:10.667 INFO:teuthology.orchestra.run.smithi152.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-22T01:54:10.742 INFO:teuthology.orchestra.run.smithi152.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-04-22T01:54:10.767 INFO:teuthology.orchestra.run.smithi152.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-04-22T01:54:10.817 INFO:teuthology.orchestra.run.smithi152.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-04-22T01:54:10.843 INFO:teuthology.orchestra.run.smithi152.stdout:(30/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.1 MB/s | 838 kB 00:00 2024-04-22T01:54:11.335 INFO:teuthology.orchestra.run.smithi152.stdout:(31/32): librbd1-19.0.0-3159.g676947e1.el9.x86_ 7.1 MB/s | 3.7 MB 00:00 2024-04-22T01:54:12.101 INFO:teuthology.orchestra.run.smithi152.stdout:(32/32): librados2-19.0.0-3159.g676947e1.el9.x8 2.5 MB/s | 3.4 MB 00:01 2024-04-22T01:54:12.103 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:54:12.104 INFO:teuthology.orchestra.run.smithi152.stdout:Total 8.9 MB/s | 59 MB 00:06 2024-04-22T01:54:12.200 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-04-22T01:54:12.237 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-04-22T01:54:12.238 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-04-22T01:54:12.697 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-04-22T01:54:12.698 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-04-22T01:54:14.252 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-04-22T01:54:14.778 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-04-22T01:54:14.838 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librados2-2:19.0.0-3159.g676947e1.el9.x86_64 2/34 2024-04-22T01:54:14.921 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:19.0.0-3159.g676947e1.el9.x86_64 2/34 2024-04-22T01:54:15.009 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rados-2:19.0.0-3159.g676947e1.el9.x86_64 3/34 2024-04-22T01:54:15.047 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs2-2:19.0.0-3159.g676947e1.el9.x86_64 4/34 2024-04-22T01:54:15.119 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs2-2:19.0.0-3159.g676947e1.el9.x86_64 4/34 2024-04-22T01:54:15.164 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : liboath-2.6.7-2.el9.x86_64 5/34 2024-04-22T01:54:15.202 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/34 2024-04-22T01:54:15.228 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/34 2024-04-22T01:54:15.250 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-22T01:54:15.274 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 8/34 2024-04-22T01:54:15.435 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-22T01:54:15.467 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 9/34 2024-04-22T01:54:15.822 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/34 2024-04-22T01:54:15.889 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librbd1-2:19.0.0-3159.g676947e1.el9.x86_64 11/34 2024-04-22T01:54:15.939 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:19.0.0-3159.g676947e1.el9.x86_64 11/34 2024-04-22T01:54:15.982 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-argparse-2:19.0.0-3159.g676947e1.el9. 12/34 2024-04-22T01:54:16.037 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cephfs-2:19.0.0-3159.g676947e1.el9.x86_64 13/34 2024-04-22T01:54:16.070 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rbd-2:19.0.0-3159.g676947e1.el9.x86_64 14/34 2024-04-22T01:54:16.159 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 15/34 2024-04-22T01:54:16.183 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libradosstriper1-2:19.0.0-3159.g676947e1.el9.x86_6 16/34 2024-04-22T01:54:16.272 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3159.g676947e1.el9.x86_6 16/34 2024-04-22T01:54:16.308 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : thrift-0.15.0-2.el9.x86_64 17/34 2024-04-22T01:54:16.419 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : re2-1:20211101-3.el9.x86_64 18/34 2024-04-22T01:54:16.458 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 19/34 2024-04-22T01:54:16.498 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 20/34 2024-04-22T01:54:16.523 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 21/34 2024-04-22T01:54:16.552 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 22/34 2024-04-22T01:54:16.583 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 23/34 2024-04-22T01:54:16.619 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-04-22T01:54:17.194 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-04-22T01:54:17.240 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librgw2-2:19.0.0-3159.g676947e1.el9.x86_64 26/34 2024-04-22T01:54:17.311 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librgw2-2:19.0.0-3159.g676947e1.el9.x86_64 26/34 2024-04-22T01:54:17.390 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rgw-2:19.0.0-3159.g676947e1.el9.x86_64 27/34 2024-04-22T01:54:17.418 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-common-2:19.0.0-3159.g676947e1.el9.x8 28/34 2024-04-22T01:54:19.851 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:19.0.0-3159.g676947e1.el9.x86_64 29/34 2024-04-22T01:54:20.031 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-common-2:19.0.0-3159.g676947e1.el9.x86_64 29/34 2024-04-22T01:54:20.697 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:19.0.0-3159.g676947e1.el9.x86_64 29/34 2024-04-22T01:54:20.735 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-base-2:19.0.0-3159.g676947e1.el9.x86_64 30/34 2024-04-22T01:54:20.815 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-base-2:19.0.0-3159.g676947e1.el9.x86_64 30/34 2024-04-22T01:54:20.815 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-22T01:54:20.815 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-22T01:54:20.816 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:54:20.836 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-selinux-2:19.0.0-3159.g676947e1.el9.x86_64 31/34 2024-04-22T01:54:30.732 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3159.g676947e1.el9.x86_64 31/34 2024-04-22T01:54:30.732 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /sys 2024-04-22T01:54:30.732 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /proc 2024-04-22T01:54:30.732 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /mnt 2024-04-22T01:54:30.732 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /var/tmp 2024-04-22T01:54:30.732 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /home 2024-04-22T01:54:30.732 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /root 2024-04-22T01:54:30.732 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /tmp 2024-04-22T01:54:30.732 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:54:31.823 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-radosgw-2:19.0.0-3159.g676947e1.el9.x86_64 32/34 2024-04-22T01:54:31.846 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3159.g676947e1.el9.x86_64 32/34 2024-04-22T01:54:31.846 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:54:31.847 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-22T01:54:31.847 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T01:54:31.847 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-22T01:54:31.847 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:54:31.848 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-22T01:54:31.934 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-04-22T01:54:31.934 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-22T01:54:33.187 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-04-22T01:54:33.187 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-base-2:19.0.0-3159.g676947e1.el9.x86_64 1/34 2024-04-22T01:54:33.187 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-common-2:19.0.0-3159.g676947e1.el9.x86_64 2/34 2024-04-22T01:54:33.187 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-radosgw-2:19.0.0-3159.g676947e1.el9.x86_64 3/34 2024-04-22T01:54:33.187 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-selinux-2:19.0.0-3159.g676947e1.el9.x86_64 4/34 2024-04-22T01:54:33.187 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs2-2:19.0.0-3159.g676947e1.el9.x86_64 5/34 2024-04-22T01:54:33.187 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libradosstriper1-2:19.0.0-3159.g676947e1.el9.x86_6 6/34 2024-04-22T01:54:33.187 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librgw2-2:19.0.0-3159.g676947e1.el9.x86_64 7/34 2024-04-22T01:54:33.187 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3159.g676947e1.el9. 8/34 2024-04-22T01:54:33.187 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-common-2:19.0.0-3159.g676947e1.el9.x8 9/34 2024-04-22T01:54:33.187 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cephfs-2:19.0.0-3159.g676947e1.el9.x86_64 10/34 2024-04-22T01:54:33.187 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rados-2:19.0.0-3159.g676947e1.el9.x86_64 11/34 2024-04-22T01:54:33.187 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rbd-2:19.0.0-3159.g676947e1.el9.x86_64 12/34 2024-04-22T01:54:33.188 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rgw-2:19.0.0-3159.g676947e1.el9.x86_64 13/34 2024-04-22T01:54:33.188 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-04-22T01:54:33.188 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-04-22T01:54:33.188 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-04-22T01:54:33.188 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-04-22T01:54:33.188 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-04-22T01:54:33.188 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-04-22T01:54:33.188 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-04-22T01:54:33.188 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-04-22T01:54:33.188 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-04-22T01:54:33.188 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-04-22T01:54:33.188 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-04-22T01:54:33.188 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-04-22T01:54:33.188 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-04-22T01:54:33.188 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-04-22T01:54:33.189 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-04-22T01:54:33.189 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-04-22T01:54:33.189 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-04-22T01:54:33.189 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:19.0.0-3159.g676947e1.el9.x86_64 31/34 2024-04-22T01:54:33.190 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-04-22T01:54:33.190 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:19.0.0-3159.g676947e1.el9.x86_64 33/34 2024-04-22T01:54:33.942 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-04-22T01:54:33.943 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:54:33.943 INFO:teuthology.orchestra.run.smithi152.stdout:Upgraded: 2024-04-22T01:54:33.943 INFO:teuthology.orchestra.run.smithi152.stdout: librados2-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:54:33.943 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:54:33.943 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-04-22T01:54:33.943 INFO:teuthology.orchestra.run.smithi152.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-22T01:54:33.943 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:54:33.943 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:54:33.943 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:54:33.943 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:54:33.943 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-22T01:54:33.944 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-22T01:54:33.944 INFO:teuthology.orchestra.run.smithi152.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-22T01:54:33.944 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:54:33.944 INFO:teuthology.orchestra.run.smithi152.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-22T01:54:33.944 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-22T01:54:33.944 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-22T01:54:33.944 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:54:33.944 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-22T01:54:33.944 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:54:33.944 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-22T01:54:33.944 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-22T01:54:33.944 INFO:teuthology.orchestra.run.smithi152.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-22T01:54:33.944 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:54:33.945 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:54:33.945 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:54:33.945 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-22T01:54:33.945 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:54:33.945 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:54:33.945 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:54:33.945 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-22T01:54:33.945 INFO:teuthology.orchestra.run.smithi152.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-22T01:54:33.945 INFO:teuthology.orchestra.run.smithi152.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-22T01:54:33.945 INFO:teuthology.orchestra.run.smithi152.stdout: re2-1:20211101-3.el9.x86_64 2024-04-22T01:54:33.945 INFO:teuthology.orchestra.run.smithi152.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-22T01:54:33.945 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:54:33.945 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:54:34.220 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-test 2024-04-22T01:54:34.831 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:31 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:54:34.958 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:54:34.959 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:54:34.959 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-04-22T01:54:34.959 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:54:34.959 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-04-22T01:54:34.959 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test x86_64 2:19.0.0-3159.g676947e1.el9 ceph 84 M 2024-04-22T01:54:34.959 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-04-22T01:54:34.959 INFO:teuthology.orchestra.run.smithi152.stdout: libasan8 x86_64 13.2.1-6.3.el9 appstream 469 k 2024-04-22T01:54:34.959 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite x86_64 2:19.0.0-3159.g676947e1.el9 ceph 174 k 2024-04-22T01:54:34.959 INFO:teuthology.orchestra.run.smithi152.stdout: libubsan x86_64 11.4.1-3.el9 appstream 187 k 2024-04-22T01:54:34.960 INFO:teuthology.orchestra.run.smithi152.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2024-04-22T01:54:34.960 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-22T01:54:34.960 INFO:teuthology.orchestra.run.smithi152.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-22T01:54:34.960 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-22T01:54:34.960 INFO:teuthology.orchestra.run.smithi152.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2024-04-22T01:54:34.960 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:54:34.960 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-04-22T01:54:34.960 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:54:34.960 INFO:teuthology.orchestra.run.smithi152.stdout:Install 9 Packages 2024-04-22T01:54:34.960 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:54:34.961 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 87 M 2024-04-22T01:54:34.961 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 812 M 2024-04-22T01:54:34.961 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-04-22T01:54:35.699 INFO:teuthology.orchestra.run.smithi152.stdout:(1/9): lksctp-tools-1.0.19-2.el9.x86_64.rpm 684 kB/s | 94 kB 00:00 2024-04-22T01:54:35.808 INFO:teuthology.orchestra.run.smithi152.stdout:(2/9): libcephsqlite-19.0.0-3159.g676947e1.el9. 703 kB/s | 174 kB 00:00 2024-04-22T01:54:35.892 INFO:teuthology.orchestra.run.smithi152.stdout:(3/9): libasan8-13.2.1-6.3.el9.x86_64.rpm 2.4 MB/s | 469 kB 00:00 2024-04-22T01:54:35.977 INFO:teuthology.orchestra.run.smithi152.stdout:(4/9): protobuf-3.14.0-13.el9.x86_64.rpm 12 MB/s | 1.0 MB 00:00 2024-04-22T01:54:36.018 INFO:teuthology.orchestra.run.smithi152.stdout:(5/9): libubsan-11.4.1-3.el9.x86_64.rpm 889 kB/s | 187 kB 00:00 2024-04-22T01:54:36.069 INFO:teuthology.orchestra.run.smithi152.stdout:(6/9): socat-1.7.4.1-5.el9.x86_64.rpm 3.2 MB/s | 305 kB 00:00 2024-04-22T01:54:36.111 INFO:teuthology.orchestra.run.smithi152.stdout:(7/9): xmlstarlet-1.6.1-20.el9.x86_64.rpm 688 kB/s | 64 kB 00:00 2024-04-22T01:54:36.314 INFO:teuthology.orchestra.run.smithi152.stdout:(8/9): yaml-cpp-0.6.3-5.el9.x86_64.rpm 497 kB/s | 122 kB 00:00 2024-04-22T01:54:44.916 INFO:teuthology.orchestra.run.smithi152.stdout:(9/9): ceph-test-19.0.0-3159.g676947e1.el9.x86_ 9.0 MB/s | 84 MB 00:09 2024-04-22T01:54:44.918 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:54:44.919 INFO:teuthology.orchestra.run.smithi152.stdout:Total 8.7 MB/s | 87 MB 00:09 2024-04-22T01:54:45.000 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-04-22T01:54:45.023 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-04-22T01:54:45.023 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-04-22T01:54:45.625 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-04-22T01:54:45.625 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-04-22T01:54:46.520 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-04-22T01:54:46.599 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/9 2024-04-22T01:54:46.651 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 2/9 2024-04-22T01:54:46.696 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 3/9 2024-04-22T01:54:46.723 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 4/9 2024-04-22T01:54:46.753 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libubsan-11.4.1-3.el9.x86_64 5/9 2024-04-22T01:54:46.778 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libasan8-13.2.1-6.3.el9.x86_64 6/9 2024-04-22T01:54:46.830 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libasan8-13.2.1-6.3.el9.x86_64 6/9 2024-04-22T01:54:46.870 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 7/9 2024-04-22T01:54:46.897 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephsqlite-2:19.0.0-3159.g676947e1.el9.x86_64 8/9 2024-04-22T01:54:58.255 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3159.g676947e1.el9.x86_64 8/9 2024-04-22T01:54:58.335 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-test-2:19.0.0-3159.g676947e1.el9.x86_64 9/9 2024-04-22T01:55:04.523 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-test-2:19.0.0-3159.g676947e1.el9.x86_64 9/9 2024-04-22T01:55:04.523 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-test-2:19.0.0-3159.g676947e1.el9.x86_64 1/9 2024-04-22T01:55:04.523 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephsqlite-2:19.0.0-3159.g676947e1.el9.x86_64 2/9 2024-04-22T01:55:04.523 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 3/9 2024-04-22T01:55:04.523 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libasan8-13.2.1-6.3.el9.x86_64 4/9 2024-04-22T01:55:04.523 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libubsan-11.4.1-3.el9.x86_64 5/9 2024-04-22T01:55:04.524 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 6/9 2024-04-22T01:55:04.524 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 7/9 2024-04-22T01:55:04.524 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 8/9 2024-04-22T01:55:04.877 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 9/9 2024-04-22T01:55:04.877 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:04.877 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-04-22T01:55:04.877 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:55:04.877 INFO:teuthology.orchestra.run.smithi152.stdout: libasan8-13.2.1-6.3.el9.x86_64 2024-04-22T01:55:04.877 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:55:04.878 INFO:teuthology.orchestra.run.smithi152.stdout: libubsan-11.4.1-3.el9.x86_64 2024-04-22T01:55:04.878 INFO:teuthology.orchestra.run.smithi152.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2024-04-22T01:55:04.878 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-22T01:55:04.878 INFO:teuthology.orchestra.run.smithi152.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-22T01:55:04.878 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-22T01:55:04.878 INFO:teuthology.orchestra.run.smithi152.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2024-04-22T01:55:04.878 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:04.878 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:55:05.148 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph 2024-04-22T01:55:05.663 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:02 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:55:05.785 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:55:05.788 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:55:05.788 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-04-22T01:55:05.788 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:55:05.788 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-04-22T01:55:05.788 INFO:teuthology.orchestra.run.smithi152.stdout: ceph x86_64 2:19.0.0-3159.g676947e1.el9 ceph 6.5 k 2024-04-22T01:55:05.788 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-04-22T01:55:05.788 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds x86_64 2:19.0.0-3159.g676947e1.el9 ceph 2.2 M 2024-04-22T01:55:05.788 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr x86_64 2:19.0.0-3159.g676947e1.el9 ceph 1.6 M 2024-04-22T01:55:05.788 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 249 k 2024-04-22T01:55:05.788 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon x86_64 2:19.0.0-3159.g676947e1.el9 ceph 5.0 M 2024-04-22T01:55:05.788 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd x86_64 2:19.0.0-3159.g676947e1.el9 ceph 71 M 2024-04-22T01:55:05.789 INFO:teuthology.orchestra.run.smithi152.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-22T01:55:05.789 INFO:teuthology.orchestra.run.smithi152.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-22T01:55:05.789 INFO:teuthology.orchestra.run.smithi152.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-22T01:55:05.789 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-22T01:55:05.789 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-22T01:55:05.789 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-22T01:55:05.789 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-22T01:55:05.789 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-22T01:55:05.789 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-22T01:55:05.789 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-22T01:55:05.789 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections 2024-04-22T01:55:05.789 INFO:teuthology.orchestra.run.smithi152.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-04-22T01:55:05.789 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-22T01:55:05.790 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-22T01:55:05.790 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-22T01:55:05.790 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-22T01:55:05.790 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-22T01:55:05.790 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-22T01:55:05.790 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-22T01:55:05.790 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-22T01:55:05.790 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-22T01:55:05.790 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-22T01:55:05.790 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-22T01:55:05.790 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-22T01:55:05.790 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-22T01:55:05.790 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-22T01:55:05.791 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-22T01:55:05.791 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-22T01:55:05.791 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-22T01:55:05.791 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-22T01:55:05.791 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-22T01:55:05.791 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:05.791 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-04-22T01:55:05.791 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:55:05.791 INFO:teuthology.orchestra.run.smithi152.stdout:Install 36 Packages 2024-04-22T01:55:05.791 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:05.792 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 84 M 2024-04-22T01:55:05.792 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 922 M 2024-04-22T01:55:05.792 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-04-22T01:55:06.674 INFO:teuthology.orchestra.run.smithi152.stdout:(1/36): ceph-19.0.0-3159.g676947e1.el9.x86_64.r 32 kB/s | 6.5 kB 00:00 2024-04-22T01:55:07.033 INFO:teuthology.orchestra.run.smithi152.stdout:(2/36): ceph-mds-19.0.0-3159.g676947e1.el9.x86_ 3.9 MB/s | 2.2 MB 00:00 2024-04-22T01:55:07.091 INFO:teuthology.orchestra.run.smithi152.stdout:(3/36): ceph-mgr-19.0.0-3159.g676947e1.el9.x86_ 2.5 MB/s | 1.6 MB 00:00 2024-04-22T01:55:07.209 INFO:teuthology.orchestra.run.smithi152.stdout:(4/36): ceph-mgr-modules-core-19.0.0-3159.g6769 2.1 MB/s | 249 kB 00:00 2024-04-22T01:55:07.450 INFO:teuthology.orchestra.run.smithi152.stdout:(5/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-04-22T01:55:07.567 INFO:teuthology.orchestra.run.smithi152.stdout:(6/36): python3-cryptography-36.0.1-4.el9.x86_6 11 MB/s | 1.2 MB 00:00 2024-04-22T01:55:07.626 INFO:teuthology.orchestra.run.smithi152.stdout:(7/36): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-22T01:55:07.734 INFO:teuthology.orchestra.run.smithi152.stdout:(8/36): ceph-mon-19.0.0-3159.g676947e1.el9.x86_ 4.7 MB/s | 5.0 MB 00:01 2024-04-22T01:55:07.759 INFO:teuthology.orchestra.run.smithi152.stdout:(9/36): python3-pycparser-2.20-6.el9.noarch.rpm 1.0 MB/s | 135 kB 00:00 2024-04-22T01:55:07.802 INFO:teuthology.orchestra.run.smithi152.stdout:(10/36): python3-requests-2.25.1-8.el9.noarch.r 1.8 MB/s | 125 kB 00:00 2024-04-22T01:55:07.902 INFO:teuthology.orchestra.run.smithi152.stdout:(11/36): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 188 kB 00:00 2024-04-22T01:55:07.969 INFO:teuthology.orchestra.run.smithi152.stdout:(12/36): python3-urllib3-1.26.5-5.el9.noarch.rp 1.0 MB/s | 215 kB 00:00 2024-04-22T01:55:08.003 INFO:teuthology.orchestra.run.smithi152.stdout:(13/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-04-22T01:55:08.020 INFO:teuthology.orchestra.run.smithi152.stdout:(14/36): python3-markupsafe-1.1.1-12.el9.x86_64 686 kB/s | 35 kB 00:00 2024-04-22T01:55:08.070 INFO:teuthology.orchestra.run.smithi152.stdout:(15/36): python3-pytz-2021.1-5.el9.noarch.rpm 761 kB/s | 51 kB 00:00 2024-04-22T01:55:08.096 INFO:teuthology.orchestra.run.smithi152.stdout:(16/36): python3-toml-0.10.2-6.el9.noarch.rpm 552 kB/s | 42 kB 00:00 2024-04-22T01:55:08.287 INFO:teuthology.orchestra.run.smithi152.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 789 kB/s | 151 kB 00:00 2024-04-22T01:55:08.363 INFO:teuthology.orchestra.run.smithi152.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 579 kB/s | 43 kB 00:00 2024-04-22T01:55:08.446 INFO:teuthology.orchestra.run.smithi152.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-04-22T01:55:08.538 INFO:teuthology.orchestra.run.smithi152.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2024-04-22T01:55:08.580 INFO:teuthology.orchestra.run.smithi152.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 256 kB/s | 11 kB 00:00 2024-04-22T01:55:08.622 INFO:teuthology.orchestra.run.smithi152.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 423 kB/s | 18 kB 00:00 2024-04-22T01:55:08.656 INFO:teuthology.orchestra.run.smithi152.stdout:(23/36): lua-devel-5.4.4-4.el9.x86_64.rpm 38 kB/s | 22 kB 00:00 2024-04-22T01:55:08.673 INFO:teuthology.orchestra.run.smithi152.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 459 kB/s | 23 kB 00:00 2024-04-22T01:55:08.723 INFO:teuthology.orchestra.run.smithi152.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 394 kB/s | 20 kB 00:00 2024-04-22T01:55:08.774 INFO:teuthology.orchestra.run.smithi152.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 166 kB/s | 19 kB 00:00 2024-04-22T01:55:08.799 INFO:teuthology.orchestra.run.smithi152.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-04-22T01:55:08.866 INFO:teuthology.orchestra.run.smithi152.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-04-22T01:55:08.891 INFO:teuthology.orchestra.run.smithi152.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 673 kB/s | 79 kB 00:00 2024-04-22T01:55:08.916 INFO:teuthology.orchestra.run.smithi152.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-04-22T01:55:08.984 INFO:teuthology.orchestra.run.smithi152.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 974 kB/s | 90 kB 00:00 2024-04-22T01:55:09.009 INFO:teuthology.orchestra.run.smithi152.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 388 kB/s | 36 kB 00:00 2024-04-22T01:55:09.084 INFO:teuthology.orchestra.run.smithi152.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-22T01:55:09.109 INFO:teuthology.orchestra.run.smithi152.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-04-22T01:55:09.135 INFO:teuthology.orchestra.run.smithi152.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2024-04-22T01:55:10.368 INFO:teuthology.orchestra.run.smithi152.stdout:(36/36): ceph-osd-19.0.0-3159.g676947e1.el9.x86 21 MB/s | 71 MB 00:03 2024-04-22T01:55:10.374 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:55:10.374 INFO:teuthology.orchestra.run.smithi152.stdout:Total 18 MB/s | 84 MB 00:04 2024-04-22T01:55:10.512 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-04-22T01:55:10.554 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-04-22T01:55:10.554 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-04-22T01:55:11.169 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-04-22T01:55:11.170 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-04-22T01:55:12.728 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-04-22T01:55:12.796 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-04-22T01:55:12.829 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-04-22T01:55:12.857 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-04-22T01:55:12.928 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-04-22T01:55:12.964 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-04-22T01:55:12.997 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-04-22T01:55:13.089 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-04-22T01:55:13.167 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-04-22T01:55:13.217 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-04-22T01:55:13.252 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-04-22T01:55:13.282 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/36 2024-04-22T01:55:13.316 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/36 2024-04-22T01:55:13.350 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/36 2024-04-22T01:55:13.390 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/36 2024-04-22T01:55:13.425 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/36 2024-04-22T01:55:13.460 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/36 2024-04-22T01:55:13.498 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/36 2024-04-22T01:55:13.572 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 18/36 2024-04-22T01:55:13.719 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 19/36 2024-04-22T01:55:13.798 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 20/36 2024-04-22T01:55:13.851 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 21/36 2024-04-22T01:55:13.890 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 22/36 2024-04-22T01:55:13.939 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ply-3.11-14.el9.noarch 23/36 2024-04-22T01:55:13.994 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 24/36 2024-04-22T01:55:14.182 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 25/36 2024-04-22T01:55:14.245 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 26/36 2024-04-22T01:55:14.481 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 27/36 2024-04-22T01:55:14.569 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 28/36 2024-04-22T01:55:14.610 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 29/36 2024-04-22T01:55:14.727 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 30/36 2024-04-22T01:55:14.921 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3159.g676947e1.el9. 31/36 2024-04-22T01:55:14.946 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-2:19.0.0-3159.g676947e1.el9.x86_64 32/36 2024-04-22T01:55:14.971 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3159.g676947e1.el9.x86_64 32/36 2024-04-22T01:55:14.971 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:55:14.971 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-22T01:55:14.971 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T01:55:14.971 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-22T01:55:14.972 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:25.746 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-osd-2:19.0.0-3159.g676947e1.el9.x86_64 33/36 2024-04-22T01:55:25.773 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-osd-2:19.0.0-3159.g676947e1.el9.x86_64 33/36 2024-04-22T01:55:25.774 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:55:25.774 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-22T01:55:25.774 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T01:55:25.774 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-22T01:55:25.774 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:26.305 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mon-2:19.0.0-3159.g676947e1.el9.x86_64 34/36 2024-04-22T01:55:26.328 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mon-2:19.0.0-3159.g676947e1.el9.x86_64 34/36 2024-04-22T01:55:26.328 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:55:26.328 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-22T01:55:26.328 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T01:55:26.328 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-22T01:55:26.328 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:26.564 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mds-2:19.0.0-3159.g676947e1.el9.x86_64 35/36 2024-04-22T01:55:26.587 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mds-2:19.0.0-3159.g676947e1.el9.x86_64 35/36 2024-04-22T01:55:26.587 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:55:26.587 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-22T01:55:26.587 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T01:55:26.587 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-22T01:55:26.587 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:26.650 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-2:19.0.0-3159.g676947e1.el9.x86_64 36/36 2024-04-22T01:55:33.525 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-2:19.0.0-3159.g676947e1.el9.x86_64 36/36 2024-04-22T01:55:33.525 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-2:19.0.0-3159.g676947e1.el9.x86_64 1/36 2024-04-22T01:55:33.525 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mds-2:19.0.0-3159.g676947e1.el9.x86_64 2/36 2024-04-22T01:55:33.525 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-2:19.0.0-3159.g676947e1.el9.x86_64 3/36 2024-04-22T01:55:33.525 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mon-2:19.0.0-3159.g676947e1.el9.x86_64 4/36 2024-04-22T01:55:33.525 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-osd-2:19.0.0-3159.g676947e1.el9.x86_64 5/36 2024-04-22T01:55:33.525 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3159.g676947e1.el9. 6/36 2024-04-22T01:55:33.525 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/36 2024-04-22T01:55:33.525 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/36 2024-04-22T01:55:33.526 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/36 2024-04-22T01:55:33.526 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/36 2024-04-22T01:55:33.526 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/36 2024-04-22T01:55:33.526 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/36 2024-04-22T01:55:33.526 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/36 2024-04-22T01:55:33.526 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/36 2024-04-22T01:55:33.526 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/36 2024-04-22T01:55:33.526 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/36 2024-04-22T01:55:33.526 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/36 2024-04-22T01:55:33.526 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-04-22T01:55:33.526 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-04-22T01:55:33.526 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-04-22T01:55:33.526 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-04-22T01:55:33.526 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-04-22T01:55:33.526 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-04-22T01:55:33.527 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-04-22T01:55:33.527 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-04-22T01:55:33.527 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-04-22T01:55:33.527 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-04-22T01:55:33.527 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-04-22T01:55:33.528 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-04-22T01:55:33.528 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-04-22T01:55:33.528 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-04-22T01:55:33.528 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-04-22T01:55:33.528 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-04-22T01:55:33.528 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-04-22T01:55:33.529 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-04-22T01:55:34.600 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-04-22T01:55:34.601 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:34.601 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-04-22T01:55:34.601 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:55:34.601 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:55:34.601 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:55:34.601 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:55:34.601 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:55:34.601 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:55:34.601 INFO:teuthology.orchestra.run.smithi152.stdout: lua-5.4.4-4.el9.x86_64 2024-04-22T01:55:34.601 INFO:teuthology.orchestra.run.smithi152.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-22T01:55:34.602 INFO:teuthology.orchestra.run.smithi152.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-22T01:55:34.602 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-22T01:55:34.602 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-22T01:55:34.602 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-22T01:55:34.602 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-22T01:55:34.602 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-22T01:55:34.602 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-22T01:55:34.602 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-22T01:55:34.602 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-22T01:55:34.602 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-22T01:55:34.602 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-22T01:55:34.602 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-22T01:55:34.602 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-22T01:55:34.602 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-22T01:55:34.603 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-22T01:55:34.603 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-22T01:55:34.603 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ply-3.11-14.el9.noarch 2024-04-22T01:55:34.603 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-22T01:55:34.603 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-22T01:55:34.603 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-22T01:55:34.603 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-22T01:55:34.603 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-22T01:55:34.603 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-22T01:55:34.603 INFO:teuthology.orchestra.run.smithi152.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-22T01:55:34.603 INFO:teuthology.orchestra.run.smithi152.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-22T01:55:34.603 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-22T01:55:34.603 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-22T01:55:34.603 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-22T01:55:34.603 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:34.604 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:55:34.889 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-base 2024-04-22T01:55:35.413 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:32 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:55:35.496 INFO:teuthology.orchestra.run.smithi152.stdout:Package ceph-base-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:55:35.541 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:55:35.542 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-04-22T01:55:35.542 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:55:35.618 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install cephadm 2024-04-22T01:55:36.130 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:33 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:55:36.254 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:55:36.254 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:55:36.255 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-04-22T01:55:36.255 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:55:36.255 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-04-22T01:55:36.255 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 762 k 2024-04-22T01:55:36.255 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:36.255 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-04-22T01:55:36.255 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:55:36.255 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-04-22T01:55:36.255 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:36.255 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 762 k 2024-04-22T01:55:36.255 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 769 k 2024-04-22T01:55:36.255 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-04-22T01:55:36.698 INFO:teuthology.orchestra.run.smithi152.stdout:cephadm-19.0.0-3159.g676947e1.el9.noarch.rpm 1.7 MB/s | 762 kB 00:00 2024-04-22T01:55:36.699 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:55:36.699 INFO:teuthology.orchestra.run.smithi152.stdout:Total 1.7 MB/s | 762 kB 00:00 2024-04-22T01:55:36.699 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-04-22T01:55:36.705 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-04-22T01:55:36.705 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-04-22T01:55:36.711 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-04-22T01:55:36.711 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-04-22T01:55:36.788 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-04-22T01:55:37.247 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:19.0.0-3159.g676947e1.el9.noarch 1/1 2024-04-22T01:55:37.326 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephadm-2:19.0.0-3159.g676947e1.el9.noarch 1/1 2024-04-22T01:55:37.695 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:19.0.0-3159.g676947e1.el9.noarch 1/1 2024-04-22T01:55:37.941 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephadm-2:19.0.0-3159.g676947e1.el9.noarch 1/1 2024-04-22T01:55:37.941 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:37.941 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-04-22T01:55:37.941 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:55:37.941 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:37.942 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:55:38.203 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-immutable-object-cache 2024-04-22T01:55:38.724 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:35 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:55:38.846 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:55:38.847 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:55:38.847 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-04-22T01:55:38.847 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:55:38.847 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-04-22T01:55:38.847 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3159.g676947e1.el9 ceph 157 k 2024-04-22T01:55:38.847 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:38.847 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-04-22T01:55:38.847 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:55:38.848 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-04-22T01:55:38.848 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:38.848 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 157 k 2024-04-22T01:55:38.848 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 640 k 2024-04-22T01:55:38.848 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-04-22T01:55:39.199 INFO:teuthology.orchestra.run.smithi152.stdout:ceph-immutable-object-cache-19.0.0-3159.g676947 448 kB/s | 157 kB 00:00 2024-04-22T01:55:39.199 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:55:39.200 INFO:teuthology.orchestra.run.smithi152.stdout:Total 446 kB/s | 157 kB 00:00 2024-04-22T01:55:39.200 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-04-22T01:55:39.210 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-04-22T01:55:39.210 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-04-22T01:55:39.268 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-04-22T01:55:39.268 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-04-22T01:55:39.442 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-04-22T01:55:39.517 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3159.g676947e1. 1/1 2024-04-22T01:55:39.541 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3159.g676947e1. 1/1 2024-04-22T01:55:39.542 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:55:39.542 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-22T01:55:39.542 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:40.436 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3159.g676947e1. 1/1 2024-04-22T01:55:40.436 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:40.436 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-04-22T01:55:40.436 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:55:40.436 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:40.437 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:55:40.625 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr 2024-04-22T01:55:41.135 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:38 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:55:41.214 INFO:teuthology.orchestra.run.smithi152.stdout:Package ceph-mgr-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:55:41.257 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:55:41.258 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-04-22T01:55:41.258 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:55:41.336 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-dashboard 2024-04-22T01:55:41.844 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:38 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:55:41.965 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:55:41.966 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:55:41.966 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-04-22T01:55:41.966 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:55:41.966 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-04-22T01:55:41.966 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 3.5 M 2024-04-22T01:55:41.967 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-04-22T01:55:41.967 INFO:teuthology.orchestra.run.smithi152.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-22T01:55:41.967 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 34 k 2024-04-22T01:55:41.967 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 16 k 2024-04-22T01:55:41.967 INFO:teuthology.orchestra.run.smithi152.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-22T01:55:41.967 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-22T01:55:41.967 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-22T01:55:41.967 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-22T01:55:41.967 INFO:teuthology.orchestra.run.smithi152.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-22T01:55:41.967 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-22T01:55:41.967 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-22T01:55:41.967 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:41.967 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-04-22T01:55:41.967 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:55:41.968 INFO:teuthology.orchestra.run.smithi152.stdout:Install 11 Packages 2024-04-22T01:55:41.968 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:41.968 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 7.6 M 2024-04-22T01:55:41.968 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 100 M 2024-04-22T01:55:41.968 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-04-22T01:55:42.571 INFO:teuthology.orchestra.run.smithi152.stdout:(1/11): ceph-prometheus-alerts-19.0.0-3159.g676 83 kB/s | 16 kB 00:00 2024-04-22T01:55:42.596 INFO:teuthology.orchestra.run.smithi152.stdout:(2/11): ceph-grafana-dashboards-19.0.0-3159.g67 153 kB/s | 34 kB 00:00 2024-04-22T01:55:42.747 INFO:teuthology.orchestra.run.smithi152.stdout:(3/11): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 269 kB 00:00 2024-04-22T01:55:42.838 INFO:teuthology.orchestra.run.smithi152.stdout:(4/11): ceph-mgr-dashboard-19.0.0-3159.g676947e 7.6 MB/s | 3.5 MB 00:00 2024-04-22T01:55:43.155 INFO:teuthology.orchestra.run.smithi152.stdout:(5/11): grpc-data-1.46.7-10.el9.noarch.rpm 66 kB/s | 19 kB 00:00 2024-04-22T01:55:43.889 INFO:teuthology.orchestra.run.smithi152.stdout:(6/11): abseil-cpp-20211102.0-3.el9.x86_64.rpm 480 kB/s | 548 kB 00:01 2024-04-22T01:55:44.089 INFO:teuthology.orchestra.run.smithi152.stdout:(7/11): python3-grpcio-tools-1.46.7-10.el9.x86_ 722 kB/s | 144 kB 00:00 2024-04-22T01:55:44.214 INFO:teuthology.orchestra.run.smithi152.stdout:(8/11): python3-repoze-lru-0.7-16.el9.noarch.rp 246 kB/s | 31 kB 00:00 2024-04-22T01:55:44.282 INFO:teuthology.orchestra.run.smithi152.stdout:(9/11): protobuf-compiler-3.14.0-13.el9.x86_64. 512 kB/s | 863 kB 00:01 2024-04-22T01:55:44.515 INFO:teuthology.orchestra.run.smithi152.stdout:(10/11): python3-routes-2.5.1-5.el9.noarch.rpm 628 kB/s | 188 kB 00:00 2024-04-22T01:55:45.915 INFO:teuthology.orchestra.run.smithi152.stdout:(11/11): python3-grpcio-1.46.7-10.el9.x86_64.rp 756 kB/s | 2.0 MB 00:02 2024-04-22T01:55:45.918 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:55:45.918 INFO:teuthology.orchestra.run.smithi152.stdout:Total 1.9 MB/s | 7.6 MB 00:03 2024-04-22T01:55:45.951 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-04-22T01:55:45.969 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-04-22T01:55:45.970 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-04-22T01:55:46.117 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-04-22T01:55:46.118 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-04-22T01:55:46.604 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-04-22T01:55:46.689 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/11 2024-04-22T01:55:46.719 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/11 2024-04-22T01:55:46.796 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 3/11 2024-04-22T01:55:46.931 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 4/11 2024-04-22T01:55:46.981 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 5/11 2024-04-22T01:55:47.071 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 6/11 2024-04-22T01:55:47.118 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 7/11 2024-04-22T01:55:47.146 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 8/11 2024-04-22T01:55:47.178 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3159.g676947e1.el9 9/11 2024-04-22T01:55:48.246 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3159.g676947e1.el 10/11 2024-04-22T01:55:48.295 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3159.g676947e1.el9.noa 11/11 2024-04-22T01:55:49.542 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3159.g676947e1.el9.noa 11/11 2024-04-22T01:55:49.542 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3159.g676947e1.el 1/11 2024-04-22T01:55:49.542 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3159.g676947e1.el9.noa 2/11 2024-04-22T01:55:49.542 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3159.g676947e1.el9 3/11 2024-04-22T01:55:49.542 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 4/11 2024-04-22T01:55:49.542 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 5/11 2024-04-22T01:55:49.542 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 6/11 2024-04-22T01:55:49.542 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 7/11 2024-04-22T01:55:49.543 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 8/11 2024-04-22T01:55:49.543 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/11 2024-04-22T01:55:49.543 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 10/11 2024-04-22T01:55:49.836 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 11/11 2024-04-22T01:55:49.836 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:49.836 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-04-22T01:55:49.836 INFO:teuthology.orchestra.run.smithi152.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-22T01:55:49.836 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:55:49.836 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:55:49.836 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:55:49.836 INFO:teuthology.orchestra.run.smithi152.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-22T01:55:49.836 INFO:teuthology.orchestra.run.smithi152.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-22T01:55:49.836 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-22T01:55:49.837 INFO:teuthology.orchestra.run.smithi152.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-22T01:55:49.837 INFO:teuthology.orchestra.run.smithi152.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-22T01:55:49.837 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-22T01:55:49.837 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-22T01:55:49.837 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:49.837 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:55:50.077 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-diskprediction-local 2024-04-22T01:55:50.581 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:47 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:55:50.703 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:55:50.704 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================= 2024-04-22T01:55:50.704 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-04-22T01:55:50.704 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================= 2024-04-22T01:55:50.704 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-04-22T01:55:50.704 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 7.4 M 2024-04-22T01:55:50.704 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-04-22T01:55:50.704 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-22T01:55:50.704 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-22T01:55:50.704 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-22T01:55:50.704 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-22T01:55:50.705 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-22T01:55:50.705 INFO:teuthology.orchestra.run.smithi152.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-22T01:55:50.705 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-22T01:55:50.705 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-22T01:55:50.705 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-22T01:55:50.705 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-22T01:55:50.705 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-22T01:55:50.705 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:50.705 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-04-22T01:55:50.706 INFO:teuthology.orchestra.run.smithi152.stdout:======================================================================================= 2024-04-22T01:55:50.706 INFO:teuthology.orchestra.run.smithi152.stdout:Install 12 Packages 2024-04-22T01:55:50.706 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:55:50.707 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 38 M 2024-04-22T01:55:50.707 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 209 M 2024-04-22T01:55:50.707 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-04-22T01:55:51.125 INFO:teuthology.orchestra.run.smithi152.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 822 kB/s | 195 kB 00:00 2024-04-22T01:55:51.175 INFO:teuthology.orchestra.run.smithi152.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 657 kB/s | 33 kB 00:00 2024-04-22T01:55:51.284 INFO:teuthology.orchestra.run.smithi152.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.0 MB/s | 803 kB 00:00 2024-04-22T01:55:51.334 INFO:teuthology.orchestra.run.smithi152.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 365 kB/s | 18 kB 00:00 2024-04-22T01:55:51.384 INFO:teuthology.orchestra.run.smithi152.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 704 kB/s | 35 kB 00:00 2024-04-22T01:55:51.476 INFO:teuthology.orchestra.run.smithi152.stdout:(6/12): ceph-mgr-diskprediction-local-19.0.0-31 13 MB/s | 7.4 MB 00:00 2024-04-22T01:55:51.677 INFO:teuthology.orchestra.run.smithi152.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.2 MB/s | 244 kB 00:00 2024-04-22T01:55:52.518 INFO:teuthology.orchestra.run.smithi152.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.2 MB/s | 3.0 MB 00:01 2024-04-22T01:55:52.644 INFO:teuthology.orchestra.run.smithi152.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 250 kB 00:00 2024-04-22T01:55:53.019 INFO:teuthology.orchestra.run.smithi152.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 2.9 MB/s | 4.7 MB 00:01 2024-04-22T01:55:53.344 INFO:teuthology.orchestra.run.smithi152.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.1 MB/s | 5.1 MB 00:01 2024-04-22T01:55:56.444 INFO:teuthology.orchestra.run.smithi152.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 4.3 MB/s | 16 MB 00:03 2024-04-22T01:55:56.445 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:55:56.446 INFO:teuthology.orchestra.run.smithi152.stdout:Total 6.6 MB/s | 38 MB 00:05 2024-04-22T01:55:56.802 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-04-22T01:55:56.828 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-04-22T01:55:56.828 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-04-22T01:55:57.236 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-04-22T01:55:57.237 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-04-22T01:55:58.076 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-04-22T01:55:58.140 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-04-22T01:55:58.192 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-04-22T01:55:58.301 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-04-22T01:55:58.352 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-04-22T01:55:58.531 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-04-22T01:55:58.559 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-04-22T01:55:58.636 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-04-22T01:55:59.292 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-04-22T01:55:59.415 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-04-22T01:56:00.635 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T01:56:01.674 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-04-22T01:56:01.704 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3159.g67694 12/12 2024-04-22T01:56:04.126 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3159.g67694 12/12 2024-04-22T01:56:04.126 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3159.g67694 1/12 2024-04-22T01:56:04.126 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-04-22T01:56:04.127 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-04-22T01:56:04.127 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-04-22T01:56:04.127 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-04-22T01:56:04.127 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-04-22T01:56:04.127 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-04-22T01:56:04.127 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-04-22T01:56:04.127 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-04-22T01:56:04.127 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-04-22T01:56:04.127 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-04-22T01:56:04.511 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-04-22T01:56:04.511 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:04.511 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-04-22T01:56:04.511 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:56:04.511 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-22T01:56:04.511 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-22T01:56:04.511 INFO:teuthology.orchestra.run.smithi152.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-22T01:56:04.511 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-22T01:56:04.511 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-22T01:56:04.512 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-22T01:56:04.512 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-22T01:56:04.512 INFO:teuthology.orchestra.run.smithi152.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-22T01:56:04.512 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-22T01:56:04.512 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-22T01:56:04.512 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-22T01:56:04.512 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:04.512 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:56:04.899 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-rook 2024-04-22T01:56:05.452 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:02 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:56:05.580 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:56:05.581 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:05.581 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-04-22T01:56:05.581 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:05.581 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-04-22T01:56:05.582 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 49 k 2024-04-22T01:56:05.582 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-04-22T01:56:05.582 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-22T01:56:05.582 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-22T01:56:05.582 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-22T01:56:05.582 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-22T01:56:05.582 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-22T01:56:05.582 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-22T01:56:05.582 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-22T01:56:05.582 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-22T01:56:05.582 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-22T01:56:05.582 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-22T01:56:05.582 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-22T01:56:05.582 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-22T01:56:05.583 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:05.583 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-04-22T01:56:05.583 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:05.583 INFO:teuthology.orchestra.run.smithi152.stdout:Install 13 Packages 2024-04-22T01:56:05.583 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:05.583 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 2.2 M 2024-04-22T01:56:05.583 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 26 M 2024-04-22T01:56:05.584 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-04-22T01:56:06.415 INFO:teuthology.orchestra.run.smithi152.stdout:(1/13): ceph-mgr-rook-19.0.0-3159.g676947e1.el9 259 kB/s | 49 kB 00:00 2024-04-22T01:56:06.756 INFO:teuthology.orchestra.run.smithi152.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 36 kB/s | 19 kB 00:00 2024-04-22T01:56:06.856 INFO:teuthology.orchestra.run.smithi152.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 41 kB/s | 26 kB 00:00 2024-04-22T01:56:07.390 INFO:teuthology.orchestra.run.smithi152.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 228 kB/s | 222 kB 00:00 2024-04-22T01:56:07.607 INFO:teuthology.orchestra.run.smithi152.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 187 kB/s | 159 kB 00:00 2024-04-22T01:56:07.832 INFO:teuthology.orchestra.run.smithi152.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 286 kB/s | 279 kB 00:00 2024-04-22T01:56:07.882 INFO:teuthology.orchestra.run.smithi152.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 117 kB/s | 32 kB 00:00 2024-04-22T01:56:08.066 INFO:teuthology.orchestra.run.smithi152.stdout:(8/13): python3-requests-oauthlib-1.3.0-12.el9. 79 kB/s | 54 kB 00:00 2024-04-22T01:56:08.091 INFO:teuthology.orchestra.run.smithi152.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 58 kB/s | 15 kB 00:00 2024-04-22T01:56:08.209 INFO:teuthology.orchestra.run.smithi152.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 622 kB/s | 203 kB 00:00 2024-04-22T01:56:08.325 INFO:teuthology.orchestra.run.smithi152.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 769 kB/s | 90 kB 00:00 2024-04-22T01:56:08.450 INFO:teuthology.orchestra.run.smithi152.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 165 kB/s | 59 kB 00:00 2024-04-22T01:56:08.617 INFO:teuthology.orchestra.run.smithi152.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.9 MB/s | 1.0 MB 00:00 2024-04-22T01:56:08.618 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:56:08.619 INFO:teuthology.orchestra.run.smithi152.stdout:Total 741 kB/s | 2.2 MB 00:03 2024-04-22T01:56:08.687 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-04-22T01:56:08.704 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-04-22T01:56:08.704 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-04-22T01:56:08.789 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-04-22T01:56:08.791 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-04-22T01:56:09.292 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-04-22T01:56:09.474 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-04-22T01:56:09.535 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-04-22T01:56:09.591 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-04-22T01:56:09.628 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-04-22T01:56:09.669 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-04-22T01:56:09.764 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-04-22T01:56:09.900 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-04-22T01:56:09.955 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-04-22T01:56:10.691 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-04-22T01:56:10.736 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-04-22T01:56:10.769 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-04-22T01:56:10.818 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-04-22T01:56:10.847 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-rook-2:19.0.0-3159.g676947e1.el9.noarch 13/13 2024-04-22T01:56:11.660 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3159.g676947e1.el9.noarch 13/13 2024-04-22T01:56:11.660 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3159.g676947e1.el9.noarch 1/13 2024-04-22T01:56:11.660 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-04-22T01:56:11.660 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-04-22T01:56:11.660 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-04-22T01:56:11.660 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-04-22T01:56:11.660 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-04-22T01:56:11.660 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-04-22T01:56:11.661 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-04-22T01:56:11.661 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-04-22T01:56:11.661 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-04-22T01:56:11.661 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-04-22T01:56:11.661 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-04-22T01:56:11.950 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-04-22T01:56:11.950 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:11.950 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-04-22T01:56:11.950 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:56:11.950 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-22T01:56:11.950 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-22T01:56:11.950 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-22T01:56:11.950 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-22T01:56:11.950 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-22T01:56:11.951 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-22T01:56:11.951 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-22T01:56:11.951 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-22T01:56:11.951 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-22T01:56:11.951 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-22T01:56:11.951 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-22T01:56:11.951 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-22T01:56:11.951 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:11.951 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:56:12.208 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-cephadm 2024-04-22T01:56:12.721 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:09 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:56:12.840 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:56:12.841 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:12.841 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-04-22T01:56:12.841 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:12.842 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-04-22T01:56:12.842 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 142 k 2024-04-22T01:56:12.842 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-04-22T01:56:12.842 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-22T01:56:12.842 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-22T01:56:12.842 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-22T01:56:12.842 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-22T01:56:12.842 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-22T01:56:12.842 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:12.842 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-04-22T01:56:12.842 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:12.842 INFO:teuthology.orchestra.run.smithi152.stdout:Install 6 Packages 2024-04-22T01:56:12.842 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:12.843 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 7.0 M 2024-04-22T01:56:12.843 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 33 M 2024-04-22T01:56:12.843 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-04-22T01:56:13.196 INFO:teuthology.orchestra.run.smithi152.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.8 MB/s | 248 kB 00:00 2024-04-22T01:56:13.280 INFO:teuthology.orchestra.run.smithi152.stdout:(2/6): ceph-mgr-cephadm-19.0.0-3159.g676947e1.e 658 kB/s | 142 kB 00:00 2024-04-22T01:56:13.380 INFO:teuthology.orchestra.run.smithi152.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 578 kB/s | 58 kB 00:00 2024-04-22T01:56:13.430 INFO:teuthology.orchestra.run.smithi152.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.3 MB/s | 548 kB 00:00 2024-04-22T01:56:13.455 INFO:teuthology.orchestra.run.smithi152.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 677 kB/s | 51 kB 00:00 2024-04-22T01:56:13.780 INFO:teuthology.orchestra.run.smithi152.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 8.3 MB/s | 6.0 MB 00:00 2024-04-22T01:56:13.781 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:56:13.782 INFO:teuthology.orchestra.run.smithi152.stdout:Total 7.4 MB/s | 7.0 MB 00:00 2024-04-22T01:56:13.851 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-04-22T01:56:13.862 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-04-22T01:56:13.862 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-04-22T01:56:13.956 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-04-22T01:56:13.959 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-04-22T01:56:14.258 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-04-22T01:56:14.428 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-04-22T01:56:14.482 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-04-22T01:56:15.005 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-04-22T01:56:15.085 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-04-22T01:56:15.149 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-04-22T01:56:15.178 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3159.g676947e1.el9.noarch 6/6 2024-04-22T01:56:15.880 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3159.g676947e1.el9.noarch 6/6 2024-04-22T01:56:15.881 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3159.g676947e1.el9.noarch 1/6 2024-04-22T01:56:15.881 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-04-22T01:56:15.881 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-04-22T01:56:15.881 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-04-22T01:56:15.881 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-04-22T01:56:16.132 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-04-22T01:56:16.133 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:16.133 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-04-22T01:56:16.133 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:56:16.133 INFO:teuthology.orchestra.run.smithi152.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-22T01:56:16.133 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-22T01:56:16.133 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-22T01:56:16.133 INFO:teuthology.orchestra.run.smithi152.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-22T01:56:16.133 INFO:teuthology.orchestra.run.smithi152.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-22T01:56:16.133 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:16.133 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:56:16.420 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-fuse 2024-04-22T01:56:16.936 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:13 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:56:17.056 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:56:17.057 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:17.057 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-04-22T01:56:17.058 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:17.058 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-04-22T01:56:17.058 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse x86_64 2:19.0.0-3159.g676947e1.el9 ceph 871 k 2024-04-22T01:56:17.058 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-04-22T01:56:17.058 INFO:teuthology.orchestra.run.smithi152.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-22T01:56:17.058 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:17.058 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-04-22T01:56:17.058 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:17.058 INFO:teuthology.orchestra.run.smithi152.stdout:Install 2 Packages 2024-04-22T01:56:17.058 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:17.058 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 951 k 2024-04-22T01:56:17.058 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 4.0 M 2024-04-22T01:56:17.058 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-04-22T01:56:17.500 INFO:teuthology.orchestra.run.smithi152.stdout:(1/2): ceph-fuse-19.0.0-3159.g676947e1.el9.x86_ 2.8 MB/s | 871 kB 00:00 2024-04-22T01:56:17.884 INFO:teuthology.orchestra.run.smithi152.stdout:(2/2): fuse-2.9.9-15.el9.x86_64.rpm 118 kB/s | 80 kB 00:00 2024-04-22T01:56:17.884 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:56:17.885 INFO:teuthology.orchestra.run.smithi152.stdout:Total 1.1 MB/s | 951 kB 00:00 2024-04-22T01:56:17.897 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-04-22T01:56:17.922 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-04-22T01:56:17.922 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-04-22T01:56:18.000 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-04-22T01:56:18.001 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-04-22T01:56:18.179 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-04-22T01:56:18.336 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-04-22T01:56:18.423 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-fuse-2:19.0.0-3159.g676947e1.el9.x86_64 2/2 2024-04-22T01:56:19.027 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-fuse-2:19.0.0-3159.g676947e1.el9.x86_64 2/2 2024-04-22T01:56:19.027 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-fuse-2:19.0.0-3159.g676947e1.el9.x86_64 1/2 2024-04-22T01:56:19.266 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-04-22T01:56:19.266 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:19.266 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-04-22T01:56:19.266 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse-2:19.0.0-3159.g676947e1.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-04-22T01:56:19.266 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:19.266 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:56:19.536 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-volume 2024-04-22T01:56:20.062 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:17 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:56:20.186 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:56:20.187 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:20.187 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-04-22T01:56:20.187 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:20.187 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-04-22T01:56:20.187 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-volume noarch 2:19.0.0-3159.g676947e1.el9 ceph-noarch 273 k 2024-04-22T01:56:20.187 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-04-22T01:56:20.187 INFO:teuthology.orchestra.run.smithi152.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-22T01:56:20.187 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:20.188 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-04-22T01:56:20.188 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:20.188 INFO:teuthology.orchestra.run.smithi152.stdout:Install 2 Packages 2024-04-22T01:56:20.188 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:20.188 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 350 k 2024-04-22T01:56:20.188 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 1.5 M 2024-04-22T01:56:20.188 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-04-22T01:56:20.473 INFO:teuthology.orchestra.run.smithi152.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 502 kB/s | 77 kB 00:00 2024-04-22T01:56:20.606 INFO:teuthology.orchestra.run.smithi152.stdout:(2/2): ceph-volume-19.0.0-3159.g676947e1.el9.no 950 kB/s | 273 kB 00:00 2024-04-22T01:56:20.607 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:56:20.607 INFO:teuthology.orchestra.run.smithi152.stdout:Total 835 kB/s | 350 kB 00:00 2024-04-22T01:56:20.618 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-04-22T01:56:20.628 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-04-22T01:56:20.628 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-04-22T01:56:20.653 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-04-22T01:56:20.654 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-04-22T01:56:20.798 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-04-22T01:56:20.988 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-04-22T01:56:21.017 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-volume-2:19.0.0-3159.g676947e1.el9.noarch 2/2 2024-04-22T01:56:21.033 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-volume-2:19.0.0-3159.g676947e1.el9.noarch 2/2 2024-04-22T01:56:21.033 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:56:21.033 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-22T01:56:21.033 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:21.687 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-volume-2:19.0.0-3159.g676947e1.el9.noarch 1/2 2024-04-22T01:56:21.949 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-04-22T01:56:21.949 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:21.949 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-04-22T01:56:21.949 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-volume-2:19.0.0-3159.g676947e1.el9.noarch 2024-04-22T01:56:21.950 INFO:teuthology.orchestra.run.smithi152.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-22T01:56:21.950 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:21.950 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:56:22.253 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librados-devel 2024-04-22T01:56:22.769 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:19 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:56:22.893 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:56:22.894 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:22.894 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-04-22T01:56:22.894 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:22.894 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-04-22T01:56:22.894 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel x86_64 2:19.0.0-3159.g676947e1.el9 ceph 125 k 2024-04-22T01:56:22.894 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:22.894 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-04-22T01:56:22.894 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:22.894 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-04-22T01:56:22.894 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:22.895 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 125 k 2024-04-22T01:56:22.895 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 445 k 2024-04-22T01:56:22.895 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-04-22T01:56:23.242 INFO:teuthology.orchestra.run.smithi152.stdout:librados-devel-19.0.0-3159.g676947e1.el9.x86_64 360 kB/s | 125 kB 00:00 2024-04-22T01:56:23.242 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:56:23.243 INFO:teuthology.orchestra.run.smithi152.stdout:Total 359 kB/s | 125 kB 00:00 2024-04-22T01:56:23.243 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-04-22T01:56:23.248 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-04-22T01:56:23.248 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-04-22T01:56:23.311 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-04-22T01:56:23.311 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-04-22T01:56:23.462 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-04-22T01:56:23.558 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librados-devel-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:56:23.990 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados-devel-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:56:24.213 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados-devel-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:56:24.213 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:24.213 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-04-22T01:56:24.213 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:56:24.214 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:24.214 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:56:24.448 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install libcephfs2 2024-04-22T01:56:24.973 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:21 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:56:25.055 INFO:teuthology.orchestra.run.smithi152.stdout:Package libcephfs2-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:56:25.097 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:56:25.098 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-04-22T01:56:25.098 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:56:25.164 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install libcephfs-devel 2024-04-22T01:56:25.680 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:22 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:56:25.800 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:56:25.801 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:25.801 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-04-22T01:56:25.801 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:25.801 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-04-22T01:56:25.801 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel x86_64 2:19.0.0-3159.g676947e1.el9 ceph 32 k 2024-04-22T01:56:25.801 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:25.801 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-04-22T01:56:25.801 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:25.801 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-04-22T01:56:25.801 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:25.801 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 32 k 2024-04-22T01:56:25.801 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 142 k 2024-04-22T01:56:25.801 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-04-22T01:56:26.043 INFO:teuthology.orchestra.run.smithi152.stdout:libcephfs-devel-19.0.0-3159.g676947e1.el9.x86_6 131 kB/s | 32 kB 00:00 2024-04-22T01:56:26.044 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:56:26.044 INFO:teuthology.orchestra.run.smithi152.stdout:Total 130 kB/s | 32 kB 00:00 2024-04-22T01:56:26.044 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-04-22T01:56:26.047 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-04-22T01:56:26.047 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-04-22T01:56:26.049 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-04-22T01:56:26.050 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-04-22T01:56:26.144 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-04-22T01:56:26.246 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs-devel-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:56:26.601 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:56:26.796 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs-devel-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:56:26.796 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:26.796 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-04-22T01:56:26.797 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:56:26.797 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:26.797 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:56:26.990 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librados2 2024-04-22T01:56:27.501 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:24 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:56:27.580 INFO:teuthology.orchestra.run.smithi152.stdout:Package librados2-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:56:27.622 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:56:27.623 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-04-22T01:56:27.623 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:56:27.694 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librbd1 2024-04-22T01:56:28.211 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:25 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:56:28.290 INFO:teuthology.orchestra.run.smithi152.stdout:Package librbd1-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:56:28.331 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:56:28.332 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-04-22T01:56:28.332 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:56:28.394 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rados 2024-04-22T01:56:28.913 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:25 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:56:28.991 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rados-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:56:29.032 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:56:29.033 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-04-22T01:56:29.033 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:56:29.095 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rgw 2024-04-22T01:56:29.612 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:26 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:56:29.692 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rgw-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:56:29.732 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:56:29.733 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-04-22T01:56:29.733 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:56:29.806 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-cephfs 2024-04-22T01:56:30.319 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:27 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:56:30.400 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-cephfs-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:56:30.441 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:56:30.442 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-04-22T01:56:30.442 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:56:30.510 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rbd 2024-04-22T01:56:31.033 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:28 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:56:31.113 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rbd-2:19.0.0-3159.g676947e1.el9.x86_64 is already installed. 2024-04-22T01:56:31.153 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:56:31.154 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2024-04-22T01:56:31.155 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:56:31.227 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-fuse 2024-04-22T01:56:31.737 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:28 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:56:31.858 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:56:31.859 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:31.859 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-04-22T01:56:31.859 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:31.859 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-04-22T01:56:31.859 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse x86_64 2:19.0.0-3159.g676947e1.el9 ceph 90 k 2024-04-22T01:56:31.859 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:31.859 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-04-22T01:56:31.859 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:31.859 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-04-22T01:56:31.859 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:31.860 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 90 k 2024-04-22T01:56:31.860 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 313 k 2024-04-22T01:56:31.860 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-04-22T01:56:32.196 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-fuse-19.0.0-3159.g676947e1.el9.x86_64.rpm 267 kB/s | 90 kB 00:00 2024-04-22T01:56:32.198 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:56:32.198 INFO:teuthology.orchestra.run.smithi152.stdout:Total 266 kB/s | 90 kB 00:00 2024-04-22T01:56:32.198 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-04-22T01:56:32.206 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-04-22T01:56:32.207 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-04-22T01:56:32.269 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-04-22T01:56:32.270 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-04-22T01:56:32.416 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-04-22T01:56:32.525 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-fuse-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:56:32.889 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-fuse-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:56:33.112 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-fuse-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:56:33.112 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:33.112 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-04-22T01:56:33.112 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:56:33.112 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:33.112 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:56:33.368 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-mirror 2024-04-22T01:56:33.894 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:30 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:56:34.016 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:56:34.016 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:34.017 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2024-04-22T01:56:34.017 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:34.017 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-04-22T01:56:34.017 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror x86_64 2:19.0.0-3159.g676947e1.el9 ceph 3.4 M 2024-04-22T01:56:34.017 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:34.017 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-04-22T01:56:34.017 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:34.017 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-04-22T01:56:34.017 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:34.017 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 3.4 M 2024-04-22T01:56:34.017 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 29 M 2024-04-22T01:56:34.017 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-04-22T01:56:34.418 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-mirror-19.0.0-3159.g676947e1.el9.x86_64.rpm 8.5 MB/s | 3.4 MB 00:00 2024-04-22T01:56:34.418 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:56:34.419 INFO:teuthology.orchestra.run.smithi152.stdout:Total 8.5 MB/s | 3.4 MB 00:00 2024-04-22T01:56:34.419 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-04-22T01:56:34.430 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-04-22T01:56:34.430 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-04-22T01:56:34.514 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-04-22T01:56:34.515 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-04-22T01:56:35.108 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-04-22T01:56:35.169 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-mirror-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:56:35.194 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:56:35.195 INFO:teuthology.orchestra.run.smithi152.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-22T01:56:35.195 INFO:teuthology.orchestra.run.smithi152.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-22T01:56:35.195 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T01:56:35.195 INFO:teuthology.orchestra.run.smithi152.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-22T01:56:35.195 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:36.438 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-mirror-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:56:36.438 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:36.438 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-04-22T01:56:36.438 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:56:36.438 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:36.438 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:56:36.681 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-nbd 2024-04-22T01:56:37.197 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:34 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:56:37.323 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:56:37.324 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:37.324 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2024-04-22T01:56:37.324 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:37.324 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-04-22T01:56:37.324 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd x86_64 2:19.0.0-3159.g676947e1.el9 ceph 175 k 2024-04-22T01:56:37.325 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:37.325 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-04-22T01:56:37.325 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:56:37.325 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2024-04-22T01:56:37.325 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:37.325 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 175 k 2024-04-22T01:56:37.325 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 673 k 2024-04-22T01:56:37.325 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-04-22T01:56:37.956 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-nbd-19.0.0-3159.g676947e1.el9.x86_64.rpm 278 kB/s | 175 kB 00:00 2024-04-22T01:56:37.957 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:56:37.957 INFO:teuthology.orchestra.run.smithi152.stdout:Total 278 kB/s | 175 kB 00:00 2024-04-22T01:56:37.957 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-04-22T01:56:37.967 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-04-22T01:56:37.967 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-04-22T01:56:38.029 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-04-22T01:56:38.030 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-04-22T01:56:38.183 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-04-22T01:56:38.286 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-nbd-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:56:38.689 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-nbd-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:56:38.929 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-nbd-2:19.0.0-3159.g676947e1.el9.x86_64 1/1 2024-04-22T01:56:38.929 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:38.929 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-04-22T01:56:38.929 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:56:38.929 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:38.929 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:56:39.176 DEBUG:teuthology.parallel:result is None 2024-04-22T01:56:39.177 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=crimson&distros=centos%2F9%2Fx86_64&sha1=676947e1f73a9d7f900df5a6854c283c4eef24a9 2024-04-22T01:56:39.323 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-22T01:56:39.343 INFO:teuthology.orchestra.run.smithi152.stdout:19.0.0-3159.g676947e1.el9 2024-04-22T01:56:39.344 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3159.g676947e1.el9 2024-04-22T01:56:39.344 INFO:teuthology.task.install:The correct ceph version 19.0.0-3159.g676947e1 is installed. 2024-04-22T01:56:39.346 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-22T01:56:39.346 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-22T01:56:39.346 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-22T01:56:39.407 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-22T01:56:39.407 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-22T01:56:39.407 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/daemon-helper 2024-04-22T01:56:39.470 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-22T01:56:39.534 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-22T01:56:39.535 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-22T01:56:39.535 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-22T01:56:39.597 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-22T01:56:39.661 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-22T01:56:39.661 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-22T01:56:39.661 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/stdin-killer 2024-04-22T01:56:39.725 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-22T01:56:39.788 INFO:teuthology.run_tasks:Running task ceph... 2024-04-22T01:56:39.879 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-22T01:56:39.879 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 777 /var/log/ceph 2024-04-22T01:56:39.903 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-22T01:56:39.903 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-22T01:56:39.980 INFO:tasks.ceph:Creating extra log directories... 2024-04-22T01:56:39.980 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-22T01:56:40.052 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-22T01:56:40.052 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-22T01:56:40.053 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668110', 'branch': 'ci-yingxin-crimson-fix-tri-mutex-3', 'description': 'crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw}', 'email': 'yingxin.cheng@intel.com', 'first_in_suite': False, 'job_id': '7668110', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'ci-yingxin-crimson-fix-tri-mutex-3'}, 'ceph': {'conf': {'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '676947e1f73a9d7f900df5a6854c283c4eef24a9'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'flavor': 'crimson', 'sha1': '676947e1f73a9d7f900df5a6854c283c4eef24a9'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'ci-yingxin-crimson-fix-tri-mutex-3', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': '676947e1f73a9d7f900df5a6854c283c4eef24a9'}}, 'owner': 'scheduled_yingxin@teuthology', 'priority': 60, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'sha1': '676947e1f73a9d7f900df5a6854c283c4eef24a9', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'ci-yingxin-crimson-fix-tri-mutex-3', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '676947e1f73a9d7f900df5a6854c283c4eef24a9', 'targets': {'smithi152.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK7vEJZAHyaYpnTOgKeOB12u7qU8K1guXN09pp+HLdP6rPm6XeYpmk9p/CF/C2mX0lNi+GqDfPWdLPJ+2dbhBvU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'global': {'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'auth supported': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': '0/0', 'debug ms': '0/0', 'debug paxos': '0/0', 'osd pool default crimson': True}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}}, 'flavor': 'crimson', 'wait-for-scrub': False, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_', '\\(OSD_', '\\(OBJECT_', 'overall HEALTH', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '676947e1f73a9d7f900df5a6854c283c4eef24a9', 'cluster': 'ceph'}}, {'cbt': {'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randrw'], 'norandommap': True, 'op_size': [4194304], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2]}}, 'cluster': {'iterations': 1, 'osds_per_node': 3, 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'user': 'ubuntu'}, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-22_01:37:40', 'tube': 'smithi', 'user': 'yingxin', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-22T01:56:40.053 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-22T01:56:40.099 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m0777 -- /var/run/ceph 2024-04-22T01:56:40.163 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-22T01:56:40.163 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T01:56:40.218 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T01:56:40.218 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_1 2024-04-22T01:56:40.273 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T01:56:40.273 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:56:40.273 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-22T01:56:40.273 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:56:40.273 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T01:56:40.273 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-04-22 01:56:35.479148058 +0000 2024-04-22T01:56:40.273 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-04-22 01:53:17.857277761 +0000 2024-04-22T01:56:40.273 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-04-22 01:53:17.857277761 +0000 2024-04-22T01:56:40.273 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-04-22 01:53:17.857277761 +0000 2024-04-22T01:56:40.274 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T01:56:40.335 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-04-22T01:56:40.335 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-04-22T01:56:40.336 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000252128 s, 2.0 MB/s 2024-04-22T01:56:40.337 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T01:56:40.393 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_2 2024-04-22T01:56:40.448 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T01:56:40.448 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:56:40.448 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-22T01:56:40.448 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:56:40.448 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T01:56:40.448 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-04-22 01:56:35.479148058 +0000 2024-04-22T01:56:40.449 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-04-22 01:53:18.211272960 +0000 2024-04-22T01:56:40.449 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-04-22 01:53:18.211272960 +0000 2024-04-22T01:56:40.449 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-04-22 01:53:18.211272960 +0000 2024-04-22T01:56:40.449 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T01:56:40.511 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-04-22T01:56:40.511 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-04-22T01:56:40.511 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000187948 s, 2.7 MB/s 2024-04-22T01:56:40.513 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T01:56:40.569 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_3 2024-04-22T01:56:40.624 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T01:56:40.624 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:56:40.624 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-22T01:56:40.624 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:56:40.624 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T01:56:40.624 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-04-22 01:56:35.479148058 +0000 2024-04-22T01:56:40.624 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-04-22 01:53:18.552268336 +0000 2024-04-22T01:56:40.624 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-04-22 01:53:18.552268336 +0000 2024-04-22T01:56:40.624 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-04-22 01:53:18.552268336 +0000 2024-04-22T01:56:40.625 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T01:56:40.686 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-04-22T01:56:40.686 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-04-22T01:56:40.686 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000182398 s, 2.8 MB/s 2024-04-22T01:56:40.687 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T01:56:40.743 DEBUG:teuthology.orchestra.run.smithi152:> stat /dev/vg_nvme/lv_4 2024-04-22T01:56:40.798 INFO:teuthology.orchestra.run.smithi152.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T01:56:40.798 INFO:teuthology.orchestra.run.smithi152.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T01:56:40.798 INFO:teuthology.orchestra.run.smithi152.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-22T01:56:40.798 INFO:teuthology.orchestra.run.smithi152.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T01:56:40.798 INFO:teuthology.orchestra.run.smithi152.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T01:56:40.798 INFO:teuthology.orchestra.run.smithi152.stdout:Access: 2024-04-22 01:56:35.480148045 +0000 2024-04-22T01:56:40.798 INFO:teuthology.orchestra.run.smithi152.stdout:Modify: 2024-04-22 01:53:18.907263521 +0000 2024-04-22T01:56:40.799 INFO:teuthology.orchestra.run.smithi152.stdout:Change: 2024-04-22 01:53:18.907263521 +0000 2024-04-22T01:56:40.799 INFO:teuthology.orchestra.run.smithi152.stdout: Birth: 2024-04-22 01:53:18.907263521 +0000 2024-04-22T01:56:40.799 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T01:56:40.860 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records in 2024-04-22T01:56:40.860 INFO:teuthology.orchestra.run.smithi152.stderr:1+0 records out 2024-04-22T01:56:40.860 INFO:teuthology.orchestra.run.smithi152.stderr:512 bytes copied, 0.000179456 s, 2.9 MB/s 2024-04-22T01:56:40.861 DEBUG:teuthology.orchestra.run.smithi152:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T01:56:40.916 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-22T01:56:40.916 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi152.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-22T01:56:40.917 INFO:tasks.ceph:Generating config... 2024-04-22T01:56:40.918 INFO:tasks.ceph:[osd] debug monc = 20 2024-04-22T01:56:40.918 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-22T01:56:40.918 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-04-22T01:56:40.918 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-04-22T01:56:40.918 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-22T01:56:40.918 INFO:tasks.ceph:[osd] bluestore rocksdb cf = False 2024-04-22T01:56:40.918 INFO:tasks.ceph:[osd] crimson alien thread cpu cores = 6-7 2024-04-22T01:56:40.918 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2024-04-22T01:56:40.918 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-22T01:56:40.918 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-22T01:56:40.918 INFO:tasks.ceph:[osd] debug filestore = 0/0 2024-04-22T01:56:40.919 INFO:tasks.ceph:[osd] debug journal = 0/0 2024-04-22T01:56:40.919 INFO:tasks.ceph:[osd] debug ms = 0/0 2024-04-22T01:56:40.919 INFO:tasks.ceph:[osd] debug osd = 0/0 2024-04-22T01:56:40.919 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-22T01:56:40.919 INFO:tasks.ceph:[osd] err to stderr = True 2024-04-22T01:56:40.919 INFO:tasks.ceph:[osd] log flush on exit = True 2024-04-22T01:56:40.919 INFO:tasks.ceph:[osd] log to file = False 2024-04-22T01:56:40.919 INFO:tasks.ceph:[osd] log to stderr = True 2024-04-22T01:56:40.919 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-22T01:56:40.919 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-22T01:56:40.919 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-22T01:56:40.919 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-22T01:56:40.919 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-22T01:56:40.919 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-04-22T01:56:40.919 INFO:tasks.ceph:[global] auth client required = none 2024-04-22T01:56:40.920 INFO:tasks.ceph:[global] auth cluster required = none 2024-04-22T01:56:40.920 INFO:tasks.ceph:[global] auth service required = none 2024-04-22T01:56:40.920 INFO:tasks.ceph:[global] auth supported = none 2024-04-22T01:56:40.920 INFO:tasks.ceph:[global] debug asok = 0/0 2024-04-22T01:56:40.920 INFO:tasks.ceph:[global] debug auth = 0/0 2024-04-22T01:56:40.920 INFO:tasks.ceph:[global] debug bdev = 0/0 2024-04-22T01:56:40.920 INFO:tasks.ceph:[global] debug bluefs = 0/0 2024-04-22T01:56:40.920 INFO:tasks.ceph:[global] debug bluestore = 0/0 2024-04-22T01:56:40.920 INFO:tasks.ceph:[global] debug buffer = 0/0 2024-04-22T01:56:40.920 INFO:tasks.ceph:[global] debug civetweb = 0/0 2024-04-22T01:56:40.920 INFO:tasks.ceph:[global] debug client = 0/0 2024-04-22T01:56:40.920 INFO:tasks.ceph:[global] debug compressor = 0/0 2024-04-22T01:56:40.920 INFO:tasks.ceph:[global] debug context = 0/0 2024-04-22T01:56:40.920 INFO:tasks.ceph:[global] debug crush = 0/0 2024-04-22T01:56:40.920 INFO:tasks.ceph:[global] debug crypto = 0/0 2024-04-22T01:56:40.920 INFO:tasks.ceph:[global] debug dpdk = 0/0 2024-04-22T01:56:40.921 INFO:tasks.ceph:[global] debug eventtrace = 0/0 2024-04-22T01:56:40.921 INFO:tasks.ceph:[global] debug filer = 0/0 2024-04-22T01:56:40.921 INFO:tasks.ceph:[global] debug filestore = 0/0 2024-04-22T01:56:40.921 INFO:tasks.ceph:[global] debug finisher = 0/0 2024-04-22T01:56:40.921 INFO:tasks.ceph:[global] debug fuse = 0/0 2024-04-22T01:56:40.921 INFO:tasks.ceph:[global] debug heartbeatmap = 0/0 2024-04-22T01:56:40.921 INFO:tasks.ceph:[global] debug javaclient = 0/0 2024-04-22T01:56:40.921 INFO:tasks.ceph:[global] debug journal = 0/0 2024-04-22T01:56:40.921 INFO:tasks.ceph:[global] debug journaler = 0/0 2024-04-22T01:56:40.921 INFO:tasks.ceph:[global] debug kstore = 0/0 2024-04-22T01:56:40.921 INFO:tasks.ceph:[global] debug leveldb = 0/0 2024-04-22T01:56:40.921 INFO:tasks.ceph:[global] debug lockdep = 0/0 2024-04-22T01:56:40.921 INFO:tasks.ceph:[global] debug mds = 0/0 2024-04-22T01:56:40.921 INFO:tasks.ceph:[global] debug mds balancer = 0/0 2024-04-22T01:56:40.921 INFO:tasks.ceph:[global] debug mds locker = 0/0 2024-04-22T01:56:40.922 INFO:tasks.ceph:[global] debug mds log = 0/0 2024-04-22T01:56:40.922 INFO:tasks.ceph:[global] debug mds log expire = 0/0 2024-04-22T01:56:40.922 INFO:tasks.ceph:[global] debug mds migrator = 0/0 2024-04-22T01:56:40.922 INFO:tasks.ceph:[global] debug memdb = 0/0 2024-04-22T01:56:40.922 INFO:tasks.ceph:[global] debug mgr = 0/0 2024-04-22T01:56:40.922 INFO:tasks.ceph:[global] debug mgrc = 0/0 2024-04-22T01:56:40.922 INFO:tasks.ceph:[global] debug mon = 0/0 2024-04-22T01:56:40.922 INFO:tasks.ceph:[global] debug monc = 0/0 2024-04-22T01:56:40.922 INFO:tasks.ceph:[global] debug ms = 0/0 2024-04-22T01:56:40.922 INFO:tasks.ceph:[global] debug objclass = 0/0 2024-04-22T01:56:40.922 INFO:tasks.ceph:[global] debug objectcacher = 0/0 2024-04-22T01:56:40.922 INFO:tasks.ceph:[global] debug objecter = 0/0 2024-04-22T01:56:40.922 INFO:tasks.ceph:[global] debug optracker = 0/0 2024-04-22T01:56:40.922 INFO:tasks.ceph:[global] debug osd = 0/0 2024-04-22T01:56:40.922 INFO:tasks.ceph:[global] debug paxos = 0/0 2024-04-22T01:56:40.923 INFO:tasks.ceph:[global] debug perfcounter = 0/0 2024-04-22T01:56:40.923 INFO:tasks.ceph:[global] debug rados = 0/0 2024-04-22T01:56:40.923 INFO:tasks.ceph:[global] debug rbd = 0/0 2024-04-22T01:56:40.923 INFO:tasks.ceph:[global] debug rbd mirror = 0/0 2024-04-22T01:56:40.923 INFO:tasks.ceph:[global] debug rbd replay = 0/0 2024-04-22T01:56:40.923 INFO:tasks.ceph:[global] debug refs = 0/0 2024-04-22T01:56:40.923 INFO:tasks.ceph:[global] debug rgw = 0/0 2024-04-22T01:56:40.923 INFO:tasks.ceph:[global] debug rgw lifecycle = 0/0 2024-04-22T01:56:40.923 INFO:tasks.ceph:[global] debug rgw notification = 0/0 2024-04-22T01:56:40.923 INFO:tasks.ceph:[global] debug rgw sync = 0/0 2024-04-22T01:56:40.923 INFO:tasks.ceph:[global] debug rocksdb = 0/0 2024-04-22T01:56:40.923 INFO:tasks.ceph:[global] debug striper = 0/0 2024-04-22T01:56:40.923 INFO:tasks.ceph:[global] debug throttle = 0/0 2024-04-22T01:56:40.923 INFO:tasks.ceph:[global] debug timer = 0/0 2024-04-22T01:56:40.923 INFO:tasks.ceph:[global] debug tp = 0/0 2024-04-22T01:56:40.924 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2024-04-22T01:56:40.924 INFO:tasks.ceph:[global] ms client mode = crc 2024-04-22T01:56:40.924 INFO:tasks.ceph:[global] ms cluster mode = crc 2024-04-22T01:56:40.924 INFO:tasks.ceph:[global] ms mon client mode = crc 2024-04-22T01:56:40.924 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2024-04-22T01:56:40.924 INFO:tasks.ceph:[global] ms mon service mode = crc 2024-04-22T01:56:40.924 INFO:tasks.ceph:[global] ms service mode = crc 2024-04-22T01:56:40.924 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-22T01:56:40.924 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-22T01:56:40.924 INFO:tasks.ceph:[mon] debug mon = 0/0 2024-04-22T01:56:40.924 INFO:tasks.ceph:[mon] debug ms = 0/0 2024-04-22T01:56:40.924 INFO:tasks.ceph:[mon] debug paxos = 0/0 2024-04-22T01:56:40.924 INFO:tasks.ceph:[mon] osd pool default crimson = True 2024-04-22T01:56:40.924 INFO:tasks.ceph:[osd.0] crimson seastar cpu cores = 0-2 2024-04-22T01:56:40.924 INFO:tasks.ceph:[osd.1] crimson seastar cpu cores = 3-5 2024-04-22T01:56:40.924 INFO:tasks.ceph:[osd.2] crimson seastar cpu cores = 0-2 2024-04-22T01:56:40.925 INFO:tasks.ceph:[osd.3] crimson seastar cpu cores = 3-5 2024-04-22T01:56:40.925 INFO:tasks.ceph:Setting up mon.a... 2024-04-22T01:56:40.925 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-22T01:56:41.008 INFO:teuthology.orchestra.run.smithi152.stdout:creating /etc/ceph/ceph.keyring 2024-04-22T01:56:41.010 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-22T01:56:41.061 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-22T01:56:41.084 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.152')] 2024-04-22T01:56:41.085 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'none', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.152', 'auth client required': 'none', 'auth cluster required': 'none', 'auth service required': 'none', 'debug asok': '0/0', 'debug auth': '0/0', 'debug bdev': '0/0', 'debug bluefs': '0/0', 'debug bluestore': '0/0', 'debug buffer': '0/0', 'debug civetweb': '0/0', 'debug client': '0/0', 'debug compressor': '0/0', 'debug context': '0/0', 'debug crush': '0/0', 'debug crypto': '0/0', 'debug dpdk': '0/0', 'debug eventtrace': '0/0', 'debug filer': '0/0', 'debug filestore': '0/0', 'debug finisher': '0/0', 'debug fuse': '0/0', 'debug heartbeatmap': '0/0', 'debug javaclient': '0/0', 'debug journal': '0/0', 'debug journaler': '0/0', 'debug kstore': '0/0', 'debug leveldb': '0/0', 'debug lockdep': '0/0', 'debug mds': '0/0', 'debug mds balancer': '0/0', 'debug mds locker': '0/0', 'debug mds log': '0/0', 'debug mds log expire': '0/0', 'debug mds migrator': '0/0', 'debug memdb': '0/0', 'debug mgr': '0/0', 'debug mgrc': '0/0', 'debug mon': '0/0', 'debug monc': '0/0', 'debug ms': '0/0', 'debug objclass': '0/0', 'debug objectcacher': '0/0', 'debug objecter': '0/0', 'debug optracker': '0/0', 'debug osd': '0/0', 'debug paxos': '0/0', 'debug perfcounter': '0/0', 'debug rados': '0/0', 'debug rbd': '0/0', 'debug rbd mirror': '0/0', 'debug rbd replay': '0/0', 'debug refs': '0/0', 'debug rgw': '0/0', 'debug rgw lifecycle': '0/0', 'debug rgw notification': '0/0', 'debug rgw sync': '0/0', 'debug rocksdb': '0/0', 'debug striper': '0/0', 'debug throttle': '0/0', 'debug timer': '0/0', 'debug tp': '0/0', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore rocksdb cf': False, 'crimson alien thread cpu cores': '6-7', 'crimson osd obc lru size': 10, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': '0/0', 'debug journal': '0/0', 'debug ms': '0/0', 'debug osd': '0/0', 'debug rocksdb': 10, 'err to stderr': True, 'log flush on exit': True, 'log to file': False, 'log to stderr': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': '0/0', 'debug mon': '0/0', 'debug paxos': '0/0', 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'osd pool default crimson': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'osd.0': {'crimson seastar cpu cores': '0-2'}, 'osd.1': {'crimson seastar cpu cores': '3-5'}, 'osd.2': {'crimson seastar cpu cores': '0-2'}, 'osd.3': {'crimson seastar cpu cores': '3-5'}} 2024-04-22T01:56:41.086 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-22T01:56:41.086 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-22T01:56:41.139 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.152 --print /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:56:41.229 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:56:41.229+0000 7fe5d9c92f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:41.229 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:56:41.229+0000 7fe5d9c92f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:41.229 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:56:41.229+0000 7fe5d9c92f00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:56:41.230 INFO:teuthology.orchestra.run.smithi152.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:56:41.230 INFO:teuthology.orchestra.run.smithi152.stdout:monmaptool: generated fsid e184d29b-933c-43c2-83b5-18ef8cadffe5 2024-04-22T01:56:41.230 INFO:teuthology.orchestra.run.smithi152.stdout:setting min_mon_release = quincy 2024-04-22T01:56:41.230 INFO:teuthology.orchestra.run.smithi152.stdout:epoch 0 2024-04-22T01:56:41.230 INFO:teuthology.orchestra.run.smithi152.stdout:fsid e184d29b-933c-43c2-83b5-18ef8cadffe5 2024-04-22T01:56:41.230 INFO:teuthology.orchestra.run.smithi152.stdout:last_changed 2024-04-22T01:56:41.231108+0000 2024-04-22T01:56:41.230 INFO:teuthology.orchestra.run.smithi152.stdout:created 2024-04-22T01:56:41.231108+0000 2024-04-22T01:56:41.230 INFO:teuthology.orchestra.run.smithi152.stdout:min_mon_release 17 (quincy) 2024-04-22T01:56:41.230 INFO:teuthology.orchestra.run.smithi152.stdout:election_strategy: 1 2024-04-22T01:56:41.230 INFO:teuthology.orchestra.run.smithi152.stdout:0: [v2:172.21.15.152:3300/0,v1:172.21.15.152:6789/0] mon.a 2024-04-22T01:56:41.231 INFO:teuthology.orchestra.run.smithi152.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2024-04-22T01:56:41.232 DEBUG:teuthology.orchestra.run.smithi152:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-22T01:56:41.246 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID e184d29b-933c-43c2-83b5-18ef8cadffe5... 2024-04-22T01:56:41.247 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-22T01:56:41.327 INFO:teuthology.orchestra.run.smithi152.stdout:[global] 2024-04-22T01:56:41.327 INFO:teuthology.orchestra.run.smithi152.stdout: chdir = "" 2024-04-22T01:56:41.327 INFO:teuthology.orchestra.run.smithi152.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-22T01:56:41.327 INFO:teuthology.orchestra.run.smithi152.stdout: auth supported = none 2024-04-22T01:56:41.327 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.327 INFO:teuthology.orchestra.run.smithi152.stdout: filestore xattr use omap = true 2024-04-22T01:56:41.327 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.328 INFO:teuthology.orchestra.run.smithi152.stdout: mon clock drift allowed = 1.000 2024-04-22T01:56:41.328 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.328 INFO:teuthology.orchestra.run.smithi152.stdout: osd crush chooseleaf type = 0 2024-04-22T01:56:41.328 INFO:teuthology.orchestra.run.smithi152.stdout: auth debug = true 2024-04-22T01:56:41.328 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.328 INFO:teuthology.orchestra.run.smithi152.stdout: ms die on old message = true 2024-04-22T01:56:41.328 INFO:teuthology.orchestra.run.smithi152.stdout: ms die on bug = true 2024-04-22T01:56:41.328 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.328 INFO:teuthology.orchestra.run.smithi152.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-22T01:56:41.328 INFO:teuthology.orchestra.run.smithi152.stdout: mon pg warn max object skew = 0 2024-04-22T01:56:41.328 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.328 INFO:teuthology.orchestra.run.smithi152.stdout: # disable pg_autoscaler by default for new pools 2024-04-22T01:56:41.328 INFO:teuthology.orchestra.run.smithi152.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-22T01:56:41.328 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.329 INFO:teuthology.orchestra.run.smithi152.stdout: osd pool default size = 2 2024-04-22T01:56:41.329 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.329 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd allow primary affinity = true 2024-04-22T01:56:41.329 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd allow pg remap = true 2024-04-22T01:56:41.329 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on legacy crush tunables = false 2024-04-22T01:56:41.329 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on crush straw calc version zero = false 2024-04-22T01:56:41.329 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on no sortbitwise = false 2024-04-22T01:56:41.329 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on osd down out interval zero = false 2024-04-22T01:56:41.329 INFO:teuthology.orchestra.run.smithi152.stdout: mon warn on too few osds = false 2024-04-22T01:56:41.329 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-22T01:56:41.329 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-22T01:56:41.329 INFO:teuthology.orchestra.run.smithi152.stdout: mon_allow_pool_size_one = true 2024-04-22T01:56:41.329 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.329 INFO:teuthology.orchestra.run.smithi152.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-22T01:56:41.329 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.330 INFO:teuthology.orchestra.run.smithi152.stdout: osd default data pool replay window = 5 2024-04-22T01:56:41.330 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.330 INFO:teuthology.orchestra.run.smithi152.stdout: mon allow pool delete = true 2024-04-22T01:56:41.330 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.330 INFO:teuthology.orchestra.run.smithi152.stdout: mon cluster log file level = debug 2024-04-22T01:56:41.330 INFO:teuthology.orchestra.run.smithi152.stdout: debug asserts on shutdown = true 2024-04-22T01:56:41.330 INFO:teuthology.orchestra.run.smithi152.stdout: mon health detail to clog = false 2024-04-22T01:56:41.330 INFO:teuthology.orchestra.run.smithi152.stdout: mon host = 172.21.15.152 2024-04-22T01:56:41.330 INFO:teuthology.orchestra.run.smithi152.stdout: auth client required = none 2024-04-22T01:56:41.330 INFO:teuthology.orchestra.run.smithi152.stdout: auth cluster required = none 2024-04-22T01:56:41.330 INFO:teuthology.orchestra.run.smithi152.stdout: auth service required = none 2024-04-22T01:56:41.330 INFO:teuthology.orchestra.run.smithi152.stdout: debug asok = 0/0 2024-04-22T01:56:41.330 INFO:teuthology.orchestra.run.smithi152.stdout: debug auth = 0/0 2024-04-22T01:56:41.330 INFO:teuthology.orchestra.run.smithi152.stdout: debug bdev = 0/0 2024-04-22T01:56:41.330 INFO:teuthology.orchestra.run.smithi152.stdout: debug bluefs = 0/0 2024-04-22T01:56:41.331 INFO:teuthology.orchestra.run.smithi152.stdout: debug bluestore = 0/0 2024-04-22T01:56:41.331 INFO:teuthology.orchestra.run.smithi152.stdout: debug buffer = 0/0 2024-04-22T01:56:41.331 INFO:teuthology.orchestra.run.smithi152.stdout: debug civetweb = 0/0 2024-04-22T01:56:41.331 INFO:teuthology.orchestra.run.smithi152.stdout: debug client = 0/0 2024-04-22T01:56:41.331 INFO:teuthology.orchestra.run.smithi152.stdout: debug compressor = 0/0 2024-04-22T01:56:41.331 INFO:teuthology.orchestra.run.smithi152.stdout: debug context = 0/0 2024-04-22T01:56:41.331 INFO:teuthology.orchestra.run.smithi152.stdout: debug crush = 0/0 2024-04-22T01:56:41.331 INFO:teuthology.orchestra.run.smithi152.stdout: debug crypto = 0/0 2024-04-22T01:56:41.331 INFO:teuthology.orchestra.run.smithi152.stdout: debug dpdk = 0/0 2024-04-22T01:56:41.331 INFO:teuthology.orchestra.run.smithi152.stdout: debug eventtrace = 0/0 2024-04-22T01:56:41.331 INFO:teuthology.orchestra.run.smithi152.stdout: debug filer = 0/0 2024-04-22T01:56:41.331 INFO:teuthology.orchestra.run.smithi152.stdout: debug filestore = 0/0 2024-04-22T01:56:41.331 INFO:teuthology.orchestra.run.smithi152.stdout: debug finisher = 0/0 2024-04-22T01:56:41.331 INFO:teuthology.orchestra.run.smithi152.stdout: debug fuse = 0/0 2024-04-22T01:56:41.332 INFO:teuthology.orchestra.run.smithi152.stdout: debug heartbeatmap = 0/0 2024-04-22T01:56:41.332 INFO:teuthology.orchestra.run.smithi152.stdout: debug javaclient = 0/0 2024-04-22T01:56:41.332 INFO:teuthology.orchestra.run.smithi152.stdout: debug journal = 0/0 2024-04-22T01:56:41.332 INFO:teuthology.orchestra.run.smithi152.stdout: debug journaler = 0/0 2024-04-22T01:56:41.332 INFO:teuthology.orchestra.run.smithi152.stdout: debug kstore = 0/0 2024-04-22T01:56:41.332 INFO:teuthology.orchestra.run.smithi152.stdout: debug leveldb = 0/0 2024-04-22T01:56:41.332 INFO:teuthology.orchestra.run.smithi152.stdout: debug lockdep = 0/0 2024-04-22T01:56:41.332 INFO:teuthology.orchestra.run.smithi152.stdout: debug mds = 0/0 2024-04-22T01:56:41.332 INFO:teuthology.orchestra.run.smithi152.stdout: debug mds balancer = 0/0 2024-04-22T01:56:41.332 INFO:teuthology.orchestra.run.smithi152.stdout: debug mds locker = 0/0 2024-04-22T01:56:41.332 INFO:teuthology.orchestra.run.smithi152.stdout: debug mds log = 0/0 2024-04-22T01:56:41.332 INFO:teuthology.orchestra.run.smithi152.stdout: debug mds log expire = 0/0 2024-04-22T01:56:41.332 INFO:teuthology.orchestra.run.smithi152.stdout: debug mds migrator = 0/0 2024-04-22T01:56:41.332 INFO:teuthology.orchestra.run.smithi152.stdout: debug memdb = 0/0 2024-04-22T01:56:41.332 INFO:teuthology.orchestra.run.smithi152.stdout: debug mgr = 0/0 2024-04-22T01:56:41.332 INFO:teuthology.orchestra.run.smithi152.stdout: debug mgrc = 0/0 2024-04-22T01:56:41.333 INFO:teuthology.orchestra.run.smithi152.stdout: debug mon = 0/0 2024-04-22T01:56:41.333 INFO:teuthology.orchestra.run.smithi152.stdout: debug monc = 0/0 2024-04-22T01:56:41.333 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 0/0 2024-04-22T01:56:41.333 INFO:teuthology.orchestra.run.smithi152.stdout: debug objclass = 0/0 2024-04-22T01:56:41.333 INFO:teuthology.orchestra.run.smithi152.stdout: debug objectcacher = 0/0 2024-04-22T01:56:41.333 INFO:teuthology.orchestra.run.smithi152.stdout: debug objecter = 0/0 2024-04-22T01:56:41.333 INFO:teuthology.orchestra.run.smithi152.stdout: debug optracker = 0/0 2024-04-22T01:56:41.333 INFO:teuthology.orchestra.run.smithi152.stdout: debug osd = 0/0 2024-04-22T01:56:41.333 INFO:teuthology.orchestra.run.smithi152.stdout: debug paxos = 0/0 2024-04-22T01:56:41.333 INFO:teuthology.orchestra.run.smithi152.stdout: debug perfcounter = 0/0 2024-04-22T01:56:41.333 INFO:teuthology.orchestra.run.smithi152.stdout: debug rados = 0/0 2024-04-22T01:56:41.333 INFO:teuthology.orchestra.run.smithi152.stdout: debug rbd = 0/0 2024-04-22T01:56:41.333 INFO:teuthology.orchestra.run.smithi152.stdout: debug rbd mirror = 0/0 2024-04-22T01:56:41.333 INFO:teuthology.orchestra.run.smithi152.stdout: debug rbd replay = 0/0 2024-04-22T01:56:41.334 INFO:teuthology.orchestra.run.smithi152.stdout: debug refs = 0/0 2024-04-22T01:56:41.334 INFO:teuthology.orchestra.run.smithi152.stdout: debug rgw = 0/0 2024-04-22T01:56:41.334 INFO:teuthology.orchestra.run.smithi152.stdout: debug rgw lifecycle = 0/0 2024-04-22T01:56:41.334 INFO:teuthology.orchestra.run.smithi152.stdout: debug rgw notification = 0/0 2024-04-22T01:56:41.334 INFO:teuthology.orchestra.run.smithi152.stdout: debug rgw sync = 0/0 2024-04-22T01:56:41.334 INFO:teuthology.orchestra.run.smithi152.stdout: debug rocksdb = 0/0 2024-04-22T01:56:41.334 INFO:teuthology.orchestra.run.smithi152.stdout: debug striper = 0/0 2024-04-22T01:56:41.334 INFO:teuthology.orchestra.run.smithi152.stdout: debug throttle = 0/0 2024-04-22T01:56:41.334 INFO:teuthology.orchestra.run.smithi152.stdout: debug timer = 0/0 2024-04-22T01:56:41.334 INFO:teuthology.orchestra.run.smithi152.stdout: debug tp = 0/0 2024-04-22T01:56:41.334 INFO:teuthology.orchestra.run.smithi152.stdout: enable experimental unrecoverable data corrupting features = crimson 2024-04-22T01:56:41.334 INFO:teuthology.orchestra.run.smithi152.stdout: ms client mode = crc 2024-04-22T01:56:41.334 INFO:teuthology.orchestra.run.smithi152.stdout: ms cluster mode = crc 2024-04-22T01:56:41.334 INFO:teuthology.orchestra.run.smithi152.stdout: ms mon client mode = crc 2024-04-22T01:56:41.335 INFO:teuthology.orchestra.run.smithi152.stdout: ms mon cluster mode = crc 2024-04-22T01:56:41.335 INFO:teuthology.orchestra.run.smithi152.stdout: ms mon service mode = crc 2024-04-22T01:56:41.335 INFO:teuthology.orchestra.run.smithi152.stdout: ms service mode = crc 2024-04-22T01:56:41.335 INFO:teuthology.orchestra.run.smithi152.stdout: fsid = e184d29b-933c-43c2-83b5-18ef8cadffe5 2024-04-22T01:56:41.335 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.335 INFO:teuthology.orchestra.run.smithi152.stdout:[osd] 2024-04-22T01:56:41.335 INFO:teuthology.orchestra.run.smithi152.stdout: osd journal size = 100 2024-04-22T01:56:41.335 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.335 INFO:teuthology.orchestra.run.smithi152.stdout: osd scrub load threshold = 5.0 2024-04-22T01:56:41.335 INFO:teuthology.orchestra.run.smithi152.stdout: osd scrub max interval = 600 2024-04-22T01:56:41.335 INFO:teuthology.orchestra.run.smithi152.stdout: osd mclock profile = high_recovery_ops 2024-04-22T01:56:41.335 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.335 INFO:teuthology.orchestra.run.smithi152.stdout: osd recover clone overlap = true 2024-04-22T01:56:41.335 INFO:teuthology.orchestra.run.smithi152.stdout: osd recovery max chunk = 1048576 2024-04-22T01:56:41.335 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.335 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug shutdown = true 2024-04-22T01:56:41.336 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug op order = true 2024-04-22T01:56:41.336 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug verify stray on activate = true 2024-04-22T01:56:41.336 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug trim objects = true 2024-04-22T01:56:41.336 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.336 INFO:teuthology.orchestra.run.smithi152.stdout: osd open classes on start = true 2024-04-22T01:56:41.336 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug pg log writeout = true 2024-04-22T01:56:41.336 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.336 INFO:teuthology.orchestra.run.smithi152.stdout: osd deep scrub update digest min age = 30 2024-04-22T01:56:41.336 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.336 INFO:teuthology.orchestra.run.smithi152.stdout: osd map max advance = 10 2024-04-22T01:56:41.336 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.336 INFO:teuthology.orchestra.run.smithi152.stdout: journal zero on create = true 2024-04-22T01:56:41.336 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.337 INFO:teuthology.orchestra.run.smithi152.stdout: filestore ondisk finisher threads = 3 2024-04-22T01:56:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: filestore apply finisher threads = 3 2024-04-22T01:56:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: bdev debug aio = true 2024-04-22T01:56:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: osd debug misdirected ops = true 2024-04-22T01:56:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: debug monc = 20 2024-04-22T01:56:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore block size = 96636764160 2024-04-22T01:56:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore compression algorithm = snappy 2024-04-22T01:56:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore compression mode = aggressive 2024-04-22T01:56:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore fsck on mount = True 2024-04-22T01:56:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: bluestore rocksdb cf = False 2024-04-22T01:56:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: crimson alien thread cpu cores = 6-7 2024-04-22T01:56:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: crimson osd obc lru size = 10 2024-04-22T01:56:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: debug bluefs = 20 2024-04-22T01:56:41.338 INFO:teuthology.orchestra.run.smithi152.stdout: debug bluestore = 20 2024-04-22T01:56:41.339 INFO:teuthology.orchestra.run.smithi152.stdout: debug filestore = 0/0 2024-04-22T01:56:41.339 INFO:teuthology.orchestra.run.smithi152.stdout: debug journal = 0/0 2024-04-22T01:56:41.339 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 0/0 2024-04-22T01:56:41.339 INFO:teuthology.orchestra.run.smithi152.stdout: debug osd = 0/0 2024-04-22T01:56:41.339 INFO:teuthology.orchestra.run.smithi152.stdout: debug rocksdb = 10 2024-04-22T01:56:41.339 INFO:teuthology.orchestra.run.smithi152.stdout: err to stderr = True 2024-04-22T01:56:41.339 INFO:teuthology.orchestra.run.smithi152.stdout: log flush on exit = True 2024-04-22T01:56:41.339 INFO:teuthology.orchestra.run.smithi152.stdout: log to file = False 2024-04-22T01:56:41.339 INFO:teuthology.orchestra.run.smithi152.stdout: log to stderr = True 2024-04-22T01:56:41.339 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd backfillfull_ratio = 0.85 2024-04-22T01:56:41.339 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd full ratio = 0.9 2024-04-22T01:56:41.339 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd nearfull ratio = 0.8 2024-04-22T01:56:41.339 INFO:teuthology.orchestra.run.smithi152.stdout: osd failsafe full ratio = 0.95 2024-04-22T01:56:41.339 INFO:teuthology.orchestra.run.smithi152.stdout: osd objectstore = bluestore 2024-04-22T01:56:41.339 INFO:teuthology.orchestra.run.smithi152.stdout: osd shutdown pgref assert = True 2024-04-22T01:56:41.340 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.340 INFO:teuthology.orchestra.run.smithi152.stdout:[mgr] 2024-04-22T01:56:41.340 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 1 2024-04-22T01:56:41.340 INFO:teuthology.orchestra.run.smithi152.stdout: debug mgr = 20 2024-04-22T01:56:41.340 INFO:teuthology.orchestra.run.smithi152.stdout: debug mon = 20 2024-04-22T01:56:41.340 INFO:teuthology.orchestra.run.smithi152.stdout: debug auth = 20 2024-04-22T01:56:41.340 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min pgs per osd = 4 2024-04-22T01:56:41.340 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min bytes per osd = 10 2024-04-22T01:56:41.340 INFO:teuthology.orchestra.run.smithi152.stdout: mgr/telemetry/nag = false 2024-04-22T01:56:41.340 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.340 INFO:teuthology.orchestra.run.smithi152.stdout:[mon] 2024-04-22T01:56:41.340 INFO:teuthology.orchestra.run.smithi152.stdout: debug ms = 0/0 2024-04-22T01:56:41.340 INFO:teuthology.orchestra.run.smithi152.stdout: debug mon = 0/0 2024-04-22T01:56:41.340 INFO:teuthology.orchestra.run.smithi152.stdout: debug paxos = 0/0 2024-04-22T01:56:41.341 INFO:teuthology.orchestra.run.smithi152.stdout: debug auth = 20 2024-04-22T01:56:41.341 INFO:teuthology.orchestra.run.smithi152.stdout: mon data avail warn = 5 2024-04-22T01:56:41.341 INFO:teuthology.orchestra.run.smithi152.stdout: mon mgr mkfs grace = 240 2024-04-22T01:56:41.341 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min pgs per osd = 4 2024-04-22T01:56:41.341 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd reporter subtree level = osd 2024-04-22T01:56:41.341 INFO:teuthology.orchestra.run.smithi152.stdout: mon osd prime pg temp = true 2024-04-22T01:56:41.341 INFO:teuthology.orchestra.run.smithi152.stdout: mon reweight min bytes per osd = 10 2024-04-22T01:56:41.341 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.341 INFO:teuthology.orchestra.run.smithi152.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-22T01:56:41.341 INFO:teuthology.orchestra.run.smithi152.stdout: auth mon ticket ttl = 660 # 11m 2024-04-22T01:56:41.341 INFO:teuthology.orchestra.run.smithi152.stdout: auth service ticket ttl = 240 # 4m 2024-04-22T01:56:41.341 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.341 INFO:teuthology.orchestra.run.smithi152.stdout: # don't complain about insecure global_id in the test suite 2024-04-22T01:56:41.341 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-22T01:56:41.341 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-22T01:56:41.342 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.342 INFO:teuthology.orchestra.run.smithi152.stdout: # 1m isn't quite enough 2024-04-22T01:56:41.342 INFO:teuthology.orchestra.run.smithi152.stdout: mon_down_mkfs_grace = 2m 2024-04-22T01:56:41.342 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.342 INFO:teuthology.orchestra.run.smithi152.stdout: mon_warn_on_filestore_osds = false 2024-04-22T01:56:41.342 INFO:teuthology.orchestra.run.smithi152.stdout: osd pool default crimson = True 2024-04-22T01:56:41.342 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:56:41.342 INFO:teuthology.orchestra.run.smithi152.stdout:[client] 2024-04-22T01:56:41.342 INFO:teuthology.orchestra.run.smithi152.stdout: rgw cache enabled = true 2024-04-22T01:56:41.342 INFO:teuthology.orchestra.run.smithi152.stdout: rgw enable ops log = true 2024-04-22T01:56:41.342 INFO:teuthology.orchestra.run.smithi152.stdout: rgw enable usage log = true 2024-04-22T01:56:41.342 INFO:teuthology.orchestra.run.smithi152.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-22T01:56:41.342 INFO:teuthology.orchestra.run.smithi152.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-22T01:56:41.342 INFO:teuthology.orchestra.run.smithi152.stdout:[mon.a] 2024-04-22T01:56:41.343 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.0] 2024-04-22T01:56:41.343 INFO:teuthology.orchestra.run.smithi152.stdout: crimson seastar cpu cores = 0-2 2024-04-22T01:56:41.343 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.1] 2024-04-22T01:56:41.343 INFO:teuthology.orchestra.run.smithi152.stdout: crimson seastar cpu cores = 3-5 2024-04-22T01:56:41.343 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.2] 2024-04-22T01:56:41.343 INFO:teuthology.orchestra.run.smithi152.stdout: crimson seastar cpu cores = 0-2 2024-04-22T01:56:41.343 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.3] 2024-04-22T01:56:41.343 INFO:teuthology.orchestra.run.smithi152.stdout: crimson seastar cpu cores = 3-5 2024-04-22T01:56:41.344 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-22T01:56:41.344 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-22T01:56:41.449 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-22T01:56:41.449 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-22T01:56:41.449 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-22T01:56:41.463 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-22T01:56:41.463 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-22T01:56:41.518 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi152.front.sepia.ceph.com 2024-04-22T01:56:41.518 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-22T01:56:41.518 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-22T01:56:41.518 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-22T01:56:41.591 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-22T01:56:41.591 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:56:41.646 INFO:tasks.ceph:Setting up mon nodes... 2024-04-22T01:56:41.646 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-22T01:56:41.646 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-22T01:56:41.744 INFO:teuthology.orchestra.run.smithi152.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-22T01:56:41.747 INFO:tasks.ceph:Setting up mds nodes... 2024-04-22T01:56:41.747 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-22T01:56:41.747 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-22T01:56:41.795 INFO:teuthology.orchestra.run.smithi152.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-22T01:56:41.807 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-22T01:56:41.807 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi152.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2024-04-22T01:56:41.807 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-22T01:56:41.830 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-22T01:56:41.831 INFO:tasks.ceph:role: osd.0 2024-04-22T01:56:41.831 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi152.front.sepia.ceph.com 2024-04-22T01:56:41.831 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-22T01:56:41.902 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T01:56:41.902 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T01:56:41.902 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T01:56:41.902 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T01:56:41.902 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T01:56:41.902 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2024-04-22T01:56:41.902 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T01:56:41.902 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T01:56:41.902 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T01:56:41.903 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T01:56:42.407 INFO:teuthology.orchestra.run.smithi152.stdout:Discarding blocks...Done. 2024-04-22T01:56:42.409 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2024-04-22T01:56:42.409 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-22T01:56:42.522 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-22T01:56:42.552 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-22T01:56:42.616 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-22T01:56:42.616 INFO:tasks.ceph:role: osd.1 2024-04-22T01:56:42.616 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi152.front.sepia.ceph.com 2024-04-22T01:56:42.616 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-22T01:56:42.687 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T01:56:42.687 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T01:56:42.687 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T01:56:42.687 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T01:56:42.687 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T01:56:42.687 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2024-04-22T01:56:42.687 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T01:56:42.688 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T01:56:42.688 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T01:56:42.688 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T01:56:43.151 INFO:teuthology.orchestra.run.smithi152.stdout:Discarding blocks...Done. 2024-04-22T01:56:43.153 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2024-04-22T01:56:43.154 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-22T01:56:43.240 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-22T01:56:43.267 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-22T01:56:43.330 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-04-22T01:56:43.330 INFO:tasks.ceph:role: osd.2 2024-04-22T01:56:43.330 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi152.front.sepia.ceph.com 2024-04-22T01:56:43.331 DEBUG:teuthology.orchestra.run.smithi152:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-22T01:56:43.401 INFO:teuthology.orchestra.run.smithi152.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-22T01:56:43.401 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-22T01:56:43.401 INFO:teuthology.orchestra.run.smithi152.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-22T01:56:43.401 INFO:teuthology.orchestra.run.smithi152.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-22T01:56:43.402 INFO:teuthology.orchestra.run.smithi152.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-22T01:56:43.402 INFO:teuthology.orchestra.run.smithi152.stdout: = sunit=0 swidth=0 blks 2024-04-22T01:56:43.402 INFO:teuthology.orchestra.run.smithi152.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-22T01:56:43.402 INFO:teuthology.orchestra.run.smithi152.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-22T01:56:43.402 INFO:teuthology.orchestra.run.smithi152.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-22T01:56:43.402 INFO:teuthology.orchestra.run.smithi152.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-22T01:56:43.895 INFO:teuthology.orchestra.run.smithi152.stdout:Discarding blocks...Done. 2024-04-22T01:56:43.897 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi152.front.sepia.ceph.com -o noatime 2024-04-22T01:56:43.898 DEBUG:teuthology.orchestra.run.smithi152:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-22T01:56:44.038 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-22T01:56:44.068 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:56:44.359 INFO:teuthology.orchestra.run.smithi152.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T01:56:44.360 INFO:teuthology.orchestra.run.smithi152.stderr:INFO 2024-04-22 01:56:44,362 seastar - Reactor backend: linux-aio 2024-04-22T01:56:44.371 INFO:teuthology.orchestra.run.smithi152.stderr:==26554==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T01:56:44.492 INFO:teuthology.orchestra.run.smithi152.stderr:DEBUG 2024-04-22 01:56:44,494 [shard 0:main] seastar - reactor::drain 2024-04-22T01:56:45.027 INFO:teuthology.orchestra.run.smithi152.stderr:----------------------------------------------------- 2024-04-22T01:56:45.027 INFO:teuthology.orchestra.run.smithi152.stderr:Suppressions used: 2024-04-22T01:56:45.027 INFO:teuthology.orchestra.run.smithi152.stderr: count bytes template 2024-04-22T01:56:45.027 INFO:teuthology.orchestra.run.smithi152.stderr: 1 8 MallocExtension::Register 2024-04-22T01:56:45.027 INFO:teuthology.orchestra.run.smithi152.stderr:----------------------------------------------------- 2024-04-22T01:56:45.027 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-04-22T01:56:45.084 INFO:teuthology.orchestra.run.smithi152.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T01:56:45.087 INFO:teuthology.orchestra.run.smithi152.stderr:INFO 2024-04-22 01:56:45,089 seastar - Reactor backend: linux-aio 2024-04-22T01:56:45.097 INFO:teuthology.orchestra.run.smithi152.stderr:==26552==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T01:56:49.763 INFO:teuthology.orchestra.run.smithi152.stderr:----------------------------------------------------- 2024-04-22T01:56:49.763 INFO:teuthology.orchestra.run.smithi152.stderr:Suppressions used: 2024-04-22T01:56:49.763 INFO:teuthology.orchestra.run.smithi152.stderr: count bytes template 2024-04-22T01:56:49.763 INFO:teuthology.orchestra.run.smithi152.stderr: 1 8 MallocExtension::Register 2024-04-22T01:56:49.763 INFO:teuthology.orchestra.run.smithi152.stderr:----------------------------------------------------- 2024-04-22T01:56:49.763 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-04-22T01:56:49.764 INFO:teuthology.orchestra.run.smithi152.stdout:created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid e184d29b-933c-43c2-83b5-18ef8cadffe5 2024-04-22T01:56:49.787 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-22T01:56:49.811 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:56:50.097 INFO:teuthology.orchestra.run.smithi152.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T01:56:50.098 INFO:teuthology.orchestra.run.smithi152.stderr:INFO 2024-04-22 01:56:50,100 seastar - Reactor backend: linux-aio 2024-04-22T01:56:50.103 INFO:teuthology.orchestra.run.smithi152.stderr:==26736==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T01:56:50.232 INFO:teuthology.orchestra.run.smithi152.stderr:DEBUG 2024-04-22 01:56:50,233 [shard 0:main] seastar - reactor::drain 2024-04-22T01:56:50.777 INFO:teuthology.orchestra.run.smithi152.stderr:----------------------------------------------------- 2024-04-22T01:56:50.777 INFO:teuthology.orchestra.run.smithi152.stderr:Suppressions used: 2024-04-22T01:56:50.777 INFO:teuthology.orchestra.run.smithi152.stderr: count bytes template 2024-04-22T01:56:50.777 INFO:teuthology.orchestra.run.smithi152.stderr: 1 8 MallocExtension::Register 2024-04-22T01:56:50.777 INFO:teuthology.orchestra.run.smithi152.stderr:----------------------------------------------------- 2024-04-22T01:56:50.777 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-04-22T01:56:50.830 INFO:teuthology.orchestra.run.smithi152.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T01:56:50.833 INFO:teuthology.orchestra.run.smithi152.stderr:INFO 2024-04-22 01:56:50,835 seastar - Reactor backend: linux-aio 2024-04-22T01:56:50.847 INFO:teuthology.orchestra.run.smithi152.stderr:==26734==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T01:56:50.938 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb98fc39 0xb862700 0xb863c7c 0xb863e92 0xb863fe8 0xb8644b1 0x3e6ef 0x36331 0x5db07f4 0x5db09d6 0x5db0bed 0x5db0ec3 0x5dff703 0x5dff909 0x5d7e3f0 0x5d7e5ca 0x5d7eb56 0x5bc3681 0x5bde745 0x6eb8f84 0x6eca757 0x6ecbbf4 0x6ecbc79 0x6ecbd7b 0x6ecbe73 0x6ecc080 0x6ecc266 0x6ecc5b5 0x6ecc959 0x6eccb89 0x6eccdf3 0x6ecd0c0 0x6ecd2de 0x6ecd396 0x6ecd45c 0x6ecd53d 0x5a2bf80 0x5a2c08a 0x5a2c146 0x5a2c202 0x5a2c2be 0x5a2c40b 0x5a2c596 0x5a2c957 0x5a48898 0xb875385 0xb88f61f 0xb93061d 0xb931c6b 0xb60e451 0xb60edcd 0x366d58a 0x2958f 0x2963f 0x3459444 2024-04-22T01:56:50.938 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-04-22T01:56:55.500 INFO:teuthology.orchestra.run.smithi152.stderr:----------------------------------------------------- 2024-04-22T01:56:55.500 INFO:teuthology.orchestra.run.smithi152.stderr:Suppressions used: 2024-04-22T01:56:55.500 INFO:teuthology.orchestra.run.smithi152.stderr: count bytes template 2024-04-22T01:56:55.500 INFO:teuthology.orchestra.run.smithi152.stderr: 1 8 MallocExtension::Register 2024-04-22T01:56:55.500 INFO:teuthology.orchestra.run.smithi152.stderr:----------------------------------------------------- 2024-04-22T01:56:55.500 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-04-22T01:56:55.501 INFO:teuthology.orchestra.run.smithi152.stdout:created object store /var/lib/ceph/osd/ceph-1 for osd.1 fsid e184d29b-933c-43c2-83b5-18ef8cadffe5 2024-04-22T01:56:55.522 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-22T01:56:55.546 DEBUG:teuthology.orchestra.run.smithi152:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:56:55.830 INFO:teuthology.orchestra.run.smithi152.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T01:56:55.831 INFO:teuthology.orchestra.run.smithi152.stderr:INFO 2024-04-22 01:56:55,833 seastar - Reactor backend: linux-aio 2024-04-22T01:56:55.836 INFO:teuthology.orchestra.run.smithi152.stderr:==26918==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T01:56:55.940 INFO:teuthology.orchestra.run.smithi152.stderr:Reactor stalled for 65 ms on shard 0. Backtrace: 0x6bddd 0xb98fc39 0xb862700 0xb863c7c 0xb863e92 0xb863fe8 0xb8644b1 0x3e6ef 0xe0e80 0x34b10c2 0x34b14be 0x34b15d7 0x5d7f16e 0x5bc3681 0x5bde745 0x6eb8f84 0x6eca757 0x6ecbbf4 0x6ecbc79 0x6ecbd7b 0x6ecbe73 0x6ecc080 0x6ecc266 0x6ecc5b5 0x6ecc959 0x6eccb89 0x6eccdf3 0x6ecd0c0 0x6ecd2de 0x6ecd396 0x6ecd45c 0x6ecd53d 0x5a2bf80 0x5a2c08a 0x5a2c146 0x5a2c202 0x5a2c2be 0x5a2c40b 0x5a2c596 0x5a2c957 0x5a48898 0xb875385 0xb88f61f 0xb93061d 0xb931c6b 0xb60e451 0xb60edcd 0x37cc91a 0x37cd473 0x366cd7a 0x2958f 0x2963f 0x3459444 2024-04-22T01:56:55.941 INFO:teuthology.orchestra.run.smithi152.stderr:kernel callstack: 2024-04-22T01:56:55.962 INFO:teuthology.orchestra.run.smithi152.stderr:DEBUG 2024-04-22 01:56:55,964 [shard 0:main] seastar - reactor::drain 2024-04-22T01:56:56.493 INFO:teuthology.orchestra.run.smithi152.stderr:----------------------------------------------------- 2024-04-22T01:56:56.494 INFO:teuthology.orchestra.run.smithi152.stderr:Suppressions used: 2024-04-22T01:56:56.494 INFO:teuthology.orchestra.run.smithi152.stderr: count bytes template 2024-04-22T01:56:56.494 INFO:teuthology.orchestra.run.smithi152.stderr: 1 8 MallocExtension::Register 2024-04-22T01:56:56.494 INFO:teuthology.orchestra.run.smithi152.stderr:----------------------------------------------------- 2024-04-22T01:56:56.494 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-04-22T01:56:56.548 INFO:teuthology.orchestra.run.smithi152.stderr:WARNING: debug mode. Not for benchmarking or production 2024-04-22T01:56:56.551 INFO:teuthology.orchestra.run.smithi152.stderr:INFO 2024-04-22 01:56:56,553 seastar - Reactor backend: linux-aio 2024-04-22T01:56:56.564 INFO:teuthology.orchestra.run.smithi152.stderr:==26916==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2024-04-22T01:57:01.196 INFO:teuthology.orchestra.run.smithi152.stderr:----------------------------------------------------- 2024-04-22T01:57:01.196 INFO:teuthology.orchestra.run.smithi152.stderr:Suppressions used: 2024-04-22T01:57:01.196 INFO:teuthology.orchestra.run.smithi152.stderr: count bytes template 2024-04-22T01:57:01.196 INFO:teuthology.orchestra.run.smithi152.stderr: 1 8 MallocExtension::Register 2024-04-22T01:57:01.196 INFO:teuthology.orchestra.run.smithi152.stderr:----------------------------------------------------- 2024-04-22T01:57:01.196 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-04-22T01:57:01.197 INFO:teuthology.orchestra.run.smithi152.stdout:created object store /var/lib/ceph/osd/ceph-2 for osd.2 fsid e184d29b-933c-43c2-83b5-18ef8cadffe5 2024-04-22T01:57:01.219 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-22T01:57:01.243 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-22T01:57:01.243 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-22T01:57:01.243 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-22T01:57:01.307 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-22T01:57:01.307 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-22T01:57:01.371 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-22T01:57:01.371 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-22T01:57:01.434 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-22T01:57:01.434 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-22T01:57:01.497 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-22T01:57:01.498 DEBUG:teuthology.orchestra.run.smithi152:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-22T01:57:01.552 INFO:tasks.ceph:Adding keys to all mons... 2024-04-22T01:57:01.552 DEBUG:teuthology.orchestra.run.smithi152:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-22T01:57:01.614 INFO:teuthology.orchestra.run.smithi152.stdout:[mgr.x] 2024-04-22T01:57:01.614 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDZwyVmjR50LBAAzyPKGyWiwrHpKDyYx+yiVA== 2024-04-22T01:57:01.614 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.0] 2024-04-22T01:57:01.614 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDdwyVmQaAjDhAAHXjdBUlPfL22V8OesoDb+Q== 2024-04-22T01:57:01.614 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.1] 2024-04-22T01:57:01.614 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDiwyVmgwoXOxAAKCyM23wBkch1TFoZGmAbWw== 2024-04-22T01:57:01.615 INFO:teuthology.orchestra.run.smithi152.stdout:[osd.2] 2024-04-22T01:57:01.615 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDowyVmRtMwKhAAGfTtDn/QfYPLCnHsOGzY2Q== 2024-04-22T01:57:01.615 INFO:teuthology.orchestra.run.smithi152.stdout:[client.0] 2024-04-22T01:57:01.615 INFO:teuthology.orchestra.run.smithi152.stdout: key = AQDZwyVmQi19LxAAVNoYXi9dVkrqP4dFbxNQHg== 2024-04-22T01:57:01.616 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-22T01:57:01.715 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T01:57:01.773 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T01:57:01.831 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-22T01:57:01.890 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-22T01:57:01.953 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-22T01:57:01.953 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-22T01:57:01.976 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-22T01:57:02.107 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:02.107+0000 7f21dd42fa80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:02.107 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:02.107+0000 7f21dd42fa80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:02.123 DEBUG:teuthology.orchestra.run.smithi152:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-22T01:57:02.146 DEBUG:teuthology.orchestra.run.smithi152:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-22T01:57:02.201 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-22T01:57:02.201 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-22T01:57:02.201 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-22T01:57:02.244 INFO:tasks.ceph.mon.a:Started 2024-04-22T01:57:02.244 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-22T01:57:02.244 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-22T01:57:02.244 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-22T01:57:02.246 INFO:tasks.ceph.mgr.x:Started 2024-04-22T01:57:02.246 INFO:tasks.ceph:Setting crush tunables to default 2024-04-22T01:57:02.247 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd crush tunables default 2024-04-22T01:57:02.326 INFO:tasks.ceph.mon.a.smithi152.stderr:2024-04-22T01:57:02.326+0000 7f9a20692a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:02.326 INFO:tasks.ceph.mon.a.smithi152.stderr:2024-04-22T01:57:02.326+0000 7f9a20692a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:02.326 INFO:tasks.ceph.mon.a.smithi152.stderr:2024-04-22T01:57:02.326+0000 7f9a20692a80 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:02.327 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:02.328+0000 7f4ada78b000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:02.357 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:02.352+0000 7f7a4255d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:02.358 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:02.359+0000 7f7a4255d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:02.427 INFO:teuthology.orchestra.run.smithi152.stderr:adjusted tunables profile to default 2024-04-22T01:57:02.439 INFO:tasks.ceph:check_enable_crimson: True 2024-04-22T01:57:02.439 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2024-04-22T01:57:02.439 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2024-04-22T01:57:02.531 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:02.531+0000 7f4ada78b000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:02.531 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:02.532+0000 7f4ada78b000 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:02.543 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:02.539+0000 7fe63ea2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:02.545 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:02.546+0000 7fe63ea2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:02.625 INFO:tasks.ceph.mon.a.smithi152.stderr:2024-04-22T01:57:02.625+0000 7f9a173f4640 -1 WARNING: experimental feature 'crimson' is enabled 2024-04-22T01:57:02.625 INFO:tasks.ceph.mon.a.smithi152.stderr:Please be aware that this feature is experimental, untested, 2024-04-22T01:57:02.625 INFO:tasks.ceph.mon.a.smithi152.stderr:unsupported, and may result in data corruption, data loss, 2024-04-22T01:57:02.625 INFO:tasks.ceph.mon.a.smithi152.stderr:and/or irreparable damage to your cluster. Do not use 2024-04-22T01:57:02.625 INFO:tasks.ceph.mon.a.smithi152.stderr:feature with important data. 2024-04-22T01:57:02.625 INFO:tasks.ceph.mon.a.smithi152.stderr: 2024-04-22T01:57:02.639 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:02.639+0000 7f4ada78b000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T01:57:02.797 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:02.797+0000 7f4ada78b000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T01:57:02.849 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:02.850+0000 7f4ada78b000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T01:57:03.095 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:03.096+0000 7f4ada78b000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T01:57:03.140 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:03.140+0000 7f4ada78b000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T01:57:03.439 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-22T01:57:03.440 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-22T01:57:03.440 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-22T01:57:03.463 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-22T01:57:03.463 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-22T01:57:03.527 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-22T01:57:03.527 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-22T01:57:03.590 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new dd3ccf06-eb54-46a9-a408-273af2563302 0 2024-04-22T01:57:03.738 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:03.733+0000 7fd1eda6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:03.740 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:03.741+0000 7fd1eda6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:03.818 INFO:teuthology.orchestra.run.smithi152.stdout:0 2024-04-22T01:57:03.830 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new 9338a6c7-7189-48cd-8bb0-ed434960d0d3 1 2024-04-22T01:57:03.867 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:03.867+0000 7f4ada78b000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T01:57:03.937 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:03.932+0000 7f529f2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:03.939 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:03.939+0000 7f529f2ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:03.961 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:03.961+0000 7f4ada78b000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T01:57:04.003 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:04.004+0000 7f4ada78b000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T01:57:04.014 INFO:teuthology.orchestra.run.smithi152.stdout:1 2024-04-22T01:57:04.026 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd new a5d13ed2-6ea0-4aa4-aff5-a4f9176e460d 2 2024-04-22T01:57:04.173 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:04.168+0000 7f769579d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:04.175 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:04.176+0000 7f769579d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:04.180 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:04.181+0000 7f4ada78b000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T01:57:04.253 INFO:teuthology.orchestra.run.smithi152.stdout:2 2024-04-22T01:57:04.261 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:04.262+0000 7f4ada78b000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T01:57:04.265 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2024-04-22T01:57:04.293 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-22T01:57:04.293 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2024-04-22T01:57:04.318 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:04.318+0000 7f4ada78b000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T01:57:04.320 INFO:tasks.ceph.osd.0:Started 2024-04-22T01:57:04.320 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2024-04-22T01:57:04.347 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-22T01:57:04.347 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2024-04-22T01:57:04.390 INFO:tasks.ceph.osd.1:Started 2024-04-22T01:57:04.390 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2024-04-22T01:57:04.419 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-22T01:57:04.419 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2024-04-22T01:57:04.442 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:04.443+0000 7f4ada78b000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T01:57:04.444 INFO:tasks.ceph.osd.2:Started 2024-04-22T01:57:04.444 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:57:04.581 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:04.577+0000 7f1ffed14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:04.584 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:04.585+0000 7f1ffed14640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:04.683 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:57:04.683 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":6,"fsid":"e184d29b-933c-43c2-83b5-18ef8cadffe5","created":"2024-04-22T01:57:02.345841+0000","modified":"2024-04-22T01:57:04.254166+0000","last_up_change":"0.000000","last_in_change":"2024-04-22T01:57:04.254166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"dd3ccf06-eb54-46a9-a408-273af2563302","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9338a6c7-7189-48cd-8bb0-ed434960d0d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d13ed2-6ea0-4aa4-aff5-a4f9176e460d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:57:04.704 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-22T01:57:04.704 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-22T01:57:04.880 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:04.881+0000 7f4ada78b000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T01:57:05.006 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-22T01:57:05.110 INFO:teuthology.misc.health.smithi152.stderr:2024-04-22T01:57:05.107+0000 7ff61b6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:05.112 INFO:teuthology.misc.health.smithi152.stderr:2024-04-22T01:57:05.113+0000 7ff61b6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:05.166 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:05.166+0000 7f4ada78b000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T01:57:05.181 INFO:teuthology.misc.health.smithi152.stdout: 2024-04-22T01:57:05.181 INFO:teuthology.misc.health.smithi152.stdout:{"epoch":6,"fsid":"e184d29b-933c-43c2-83b5-18ef8cadffe5","created":"2024-04-22T01:57:02.345841+0000","modified":"2024-04-22T01:57:04.254166+0000","last_up_change":"0.000000","last_in_change":"2024-04-22T01:57:04.254166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"dd3ccf06-eb54-46a9-a408-273af2563302","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9338a6c7-7189-48cd-8bb0-ed434960d0d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d13ed2-6ea0-4aa4-aff5-a4f9176e460d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:57:05.193 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-04-22T01:57:05.225 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:05.225+0000 7f4ada78b000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T01:57:05.354 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:05.355+0000 7f4ada78b000 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-22T01:57:05.658 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:05.658+0000 7f4ada78b000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T01:57:05.710 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:05.710+0000 7f4ada78b000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T01:57:05.879 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:05.880+0000 7f4ada78b000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T01:57:06.016 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:06.017+0000 7f4ada78b000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T01:57:06.069 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:06.069+0000 7f4ada78b000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T01:57:06.142 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:06.142+0000 7f4ada78b000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T01:57:06.189 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:06.190+0000 7f4ada78b000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T01:57:06.700 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:06.701+0000 7f4ada78b000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T01:57:06.834 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:06.835+0000 7f4ada78b000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T01:57:07.844 INFO:tasks.ceph.mgr.x.smithi152.stderr:2024-04-22T01:57:07.845+0000 7f4a7991c640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-22T01:57:11.496 DEBUG:teuthology.orchestra.run.smithi152:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-22T01:57:11.603 INFO:teuthology.misc.health.smithi152.stderr:2024-04-22T01:57:11.600+0000 7f1f60e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:11.605 INFO:teuthology.misc.health.smithi152.stderr:2024-04-22T01:57:11.606+0000 7f1f60e64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:11.796 INFO:teuthology.misc.health.smithi152.stdout: 2024-04-22T01:57:11.796 INFO:teuthology.misc.health.smithi152.stdout:{"epoch":9,"fsid":"e184d29b-933c-43c2-83b5-18ef8cadffe5","created":"2024-04-22T01:57:02.345841+0000","modified":"2024-04-22T01:57:10.851782+0000","last_up_change":"2024-04-22T01:57:10.851782+0000","last_in_change":"2024-04-22T01:57:04.254166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"dd3ccf06-eb54-46a9-a408-273af2563302","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3230096197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6809","nonce":3230096197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6807","nonce":3230096197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3230096197}]},"public_addr":"172.21.15.152:6808/3230096197","cluster_addr":"172.21.15.152:6809/3230096197","heartbeat_back_addr":"172.21.15.152:6807/3230096197","heartbeat_front_addr":"172.21.15.152:6806/3230096197","state":["exists","up"]},{"osd":1,"uuid":"9338a6c7-7189-48cd-8bb0-ed434960d0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2061527554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6805","nonce":2061527554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6803","nonce":2061527554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2061527554}]},"public_addr":"172.21.15.152:6804/2061527554","cluster_addr":"172.21.15.152:6805/2061527554","heartbeat_back_addr":"172.21.15.152:6803/2061527554","heartbeat_front_addr":"172.21.15.152:6802/2061527554","state":["exists","up"]},{"osd":2,"uuid":"a5d13ed2-6ea0-4aa4-aff5-a4f9176e460d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6813","nonce":1488133494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1488133494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1488133494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6811","nonce":1488133494}]},"public_addr":"172.21.15.152:6813/1488133494","cluster_addr":"172.21.15.152:6812/1488133494","heartbeat_back_addr":"172.21.15.152:6810/1488133494","heartbeat_front_addr":"172.21.15.152:6811/1488133494","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:57:11.808 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-04-22T01:57:11.808 INFO:tasks.ceph:Creating RBD pool 2024-04-22T01:57:11.808 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd pool create rbd 8 2024-04-22T01:57:11.922 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:11.916+0000 7f5db506d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:11.923 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:11.924+0000 7f5db506d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:12.852 INFO:teuthology.orchestra.run.smithi152.stderr:pool 'rbd' created 2024-04-22T01:57:12.872 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-04-22T01:57:13.006 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:12.996+0000 7ff1ccc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:13.009 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:13.010+0000 7ff1ccc2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:13.864 INFO:teuthology.orchestra.run.smithi152.stderr:enabled application 'rbd' on pool 'rbd' 2024-04-22T01:57:13.885 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-22T01:57:13.886 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-22T01:57:13.886 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-22T01:57:13.886 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-22T01:57:13.886 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-22T01:57:14.046 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:14.043+0000 7f95e346d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:14.048 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:14.049+0000 7f95e346d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:14.283 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:57:14.295 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":4,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6800","nonce":1768672885},{"type":"v1","addr":"172.21.15.152:6801","nonce":1768672885}]},"active_addr":"172.21.15.152:6801/1768672885","active_change":"2024-04-22T01:57:06.837568+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.152:0","nonce":3373796731}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.152:0","nonce":2801301411}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.152:0","nonce":4199219259}]}]} 2024-04-22T01:57:14.299 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-22T01:57:14.299 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-22T01:57:14.299 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:57:14.407 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:14.404+0000 7f899da6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:14.408 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:14.409+0000 7f899da6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:14.595 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:57:14.596 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":12,"fsid":"e184d29b-933c-43c2-83b5-18ef8cadffe5","created":"2024-04-22T01:57:02.345841+0000","modified":"2024-04-22T01:57:13.863532+0000","last_up_change":"2024-04-22T01:57:10.851782+0000","last_in_change":"2024-04-22T01:57:04.254166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:57:10.861902+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-22T01:57:12.113744+0000","flags":294945,"flags_names":"hashpspool,nopgchange,creating,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dd3ccf06-eb54-46a9-a408-273af2563302","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3230096197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6809","nonce":3230096197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6807","nonce":3230096197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3230096197}]},"public_addr":"172.21.15.152:6808/3230096197","cluster_addr":"172.21.15.152:6809/3230096197","heartbeat_back_addr":"172.21.15.152:6807/3230096197","heartbeat_front_addr":"172.21.15.152:6806/3230096197","state":["exists","up"]},{"osd":1,"uuid":"9338a6c7-7189-48cd-8bb0-ed434960d0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2061527554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6805","nonce":2061527554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6803","nonce":2061527554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2061527554}]},"public_addr":"172.21.15.152:6804/2061527554","cluster_addr":"172.21.15.152:6805/2061527554","heartbeat_back_addr":"172.21.15.152:6803/2061527554","heartbeat_front_addr":"172.21.15.152:6802/2061527554","state":["exists","up"]},{"osd":2,"uuid":"a5d13ed2-6ea0-4aa4-aff5-a4f9176e460d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6813","nonce":1488133494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1488133494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1488133494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6811","nonce":1488133494}]},"public_addr":"172.21.15.152:6813/1488133494","cluster_addr":"172.21.15.152:6812/1488133494","heartbeat_back_addr":"172.21.15.152:6810/1488133494","heartbeat_front_addr":"172.21.15.152:6811/1488133494","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:57:14.609 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-22T01:57:14.609 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-22T01:57:14.717 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:14.714+0000 7fc907aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:14.719 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:14.720+0000 7fc907aed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:14.910 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:57:14.910 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":13,"fsid":"e184d29b-933c-43c2-83b5-18ef8cadffe5","created":"2024-04-22T01:57:02.345841+0000","modified":"2024-04-22T01:57:14.866918+0000","last_up_change":"2024-04-22T01:57:10.851782+0000","last_in_change":"2024-04-22T01:57:04.254166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":true,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T01:57:10.861902+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-04-22T01:57:12.113744+0000","flags":262177,"flags_names":"hashpspool,nopgchange,crimson","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"dd3ccf06-eb54-46a9-a408-273af2563302","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6808","nonce":3230096197}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6809","nonce":3230096197}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6807","nonce":3230096197}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6806","nonce":3230096197}]},"public_addr":"172.21.15.152:6808/3230096197","cluster_addr":"172.21.15.152:6809/3230096197","heartbeat_back_addr":"172.21.15.152:6807/3230096197","heartbeat_front_addr":"172.21.15.152:6806/3230096197","state":["exists","up"]},{"osd":1,"uuid":"9338a6c7-7189-48cd-8bb0-ed434960d0d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6804","nonce":2061527554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6805","nonce":2061527554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6803","nonce":2061527554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6802","nonce":2061527554}]},"public_addr":"172.21.15.152:6804/2061527554","cluster_addr":"172.21.15.152:6805/2061527554","heartbeat_back_addr":"172.21.15.152:6803/2061527554","heartbeat_front_addr":"172.21.15.152:6802/2061527554","state":["exists","up"]},{"osd":2,"uuid":"a5d13ed2-6ea0-4aa4-aff5-a4f9176e460d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6813","nonce":1488133494}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6812","nonce":1488133494}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6810","nonce":1488133494}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.152:6811","nonce":1488133494}]},"public_addr":"172.21.15.152:6813/1488133494","cluster_addr":"172.21.15.152:6812/1488133494","heartbeat_back_addr":"172.21.15.152:6810/1488133494","heartbeat_front_addr":"172.21.15.152:6811/1488133494","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T01:57:14.924 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-22T01:57:14.924 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-22T01:57:14.924 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-22T01:57:15.038 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:15.034+0000 7fbd22a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:15.039 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:15.034+0000 7f0b708ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:15.039 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:15.035+0000 7fb473f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:15.041 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:15.041+0000 7fb473f9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:15.041 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:15.042+0000 7fbd22a2d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:15.041 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:15.042+0000 7f0b708ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:15.071 INFO:teuthology.orchestra.run.smithi152.stdout:34359738373 2024-04-22T01:57:15.071 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T01:57:15.074 INFO:teuthology.orchestra.run.smithi152.stdout:34359738373 2024-04-22T01:57:15.074 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T01:57:15.075 INFO:teuthology.orchestra.run.smithi152.stdout:38654705668 2024-04-22T01:57:15.075 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T01:57:15.182 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:15.178+0000 7f1ee1ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:15.185 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:15.186+0000 7f1ee1ead640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:15.227 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:15.223+0000 7f238cc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:15.227 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:15.223+0000 7f4afba6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:15.229 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:15.230+0000 7f238cc6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:15.230 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:15.231+0000 7f4afba6d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:15.413 INFO:teuthology.orchestra.run.smithi152.stdout:0 2024-04-22T01:57:15.422 INFO:teuthology.orchestra.run.smithi152.stdout:0 2024-04-22T01:57:15.426 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.1 2024-04-22T01:57:15.435 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.0 2024-04-22T01:57:15.457 INFO:teuthology.orchestra.run.smithi152.stdout:0 2024-04-22T01:57:15.470 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.2 2024-04-22T01:57:16.427 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T01:57:16.436 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T01:57:16.470 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T01:57:16.541 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:16.537+0000 7fbcf70ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:16.545 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:16.546+0000 7fbcf70ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:16.584 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:16.580+0000 7fbeea6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:16.586 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:16.587+0000 7fbeea6ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:16.587 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:16.583+0000 7f6e084ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:16.596 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:16.596+0000 7f6e084ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:16.774 INFO:teuthology.orchestra.run.smithi152.stdout:0 2024-04-22T01:57:16.782 INFO:teuthology.orchestra.run.smithi152.stdout:0 2024-04-22T01:57:16.787 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.1 2024-04-22T01:57:16.795 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.2 2024-04-22T01:57:16.809 INFO:teuthology.orchestra.run.smithi152.stdout:0 2024-04-22T01:57:16.822 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.0 2024-04-22T01:57:17.787 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T01:57:17.796 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T01:57:17.823 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T01:57:17.915 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:17.908+0000 7fd240d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:17.917 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:17.918+0000 7fd240d9d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:17.949 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:17.945+0000 7f84c686d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:17.950 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:17.945+0000 7f8466b54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:17.951 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:17.952+0000 7f8466b54640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:17.952 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:17.953+0000 7f84c686d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:18.132 INFO:teuthology.orchestra.run.smithi152.stdout:0 2024-04-22T01:57:18.146 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.1 2024-04-22T01:57:18.146 INFO:teuthology.orchestra.run.smithi152.stdout:0 2024-04-22T01:57:18.158 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.2 2024-04-22T01:57:18.173 INFO:teuthology.orchestra.run.smithi152.stdout:0 2024-04-22T01:57:18.187 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.0 2024-04-22T01:57:19.147 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T01:57:19.160 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T01:57:19.187 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T01:57:19.263 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:19.259+0000 7f8d7d15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:19.267 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:19.268+0000 7f8d7d15d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:19.306 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:19.302+0000 7faa16a13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:19.308 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:19.304+0000 7feaeb824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:19.309 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:19.309+0000 7faa16a13640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:19.311 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:19.312+0000 7feaeb824640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:19.472 INFO:teuthology.orchestra.run.smithi152.stdout:0 2024-04-22T01:57:19.486 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.1 2024-04-22T01:57:19.511 INFO:teuthology.orchestra.run.smithi152.stdout:0 2024-04-22T01:57:19.518 INFO:teuthology.orchestra.run.smithi152.stdout:0 2024-04-22T01:57:19.525 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 0 for osd.2 2024-04-22T01:57:19.532 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 0 for osd.0 2024-04-22T01:57:20.487 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-22T01:57:20.526 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-22T01:57:20.533 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-22T01:57:20.606 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:20.601+0000 7f94c999d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:20.608 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:20.609+0000 7f94c999d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:20.647 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:20.642+0000 7f4efc26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:20.649 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:20.650+0000 7f4efc26d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:20.687 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:20.683+0000 7efc8a19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:20.689 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:20.689+0000 7efc8a19d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:20.818 INFO:teuthology.orchestra.run.smithi152.stdout:34359738377 2024-04-22T01:57:20.859 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738377 for osd.1 2024-04-22T01:57:20.859 DEBUG:teuthology.parallel:result is None 2024-04-22T01:57:20.866 INFO:teuthology.orchestra.run.smithi152.stdout:38654705669 2024-04-22T01:57:20.881 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705669 for osd.2 2024-04-22T01:57:20.881 DEBUG:teuthology.parallel:result is None 2024-04-22T01:57:20.896 INFO:teuthology.orchestra.run.smithi152.stdout:34359738377 2024-04-22T01:57:20.909 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738377 for osd.0 2024-04-22T01:57:20.909 DEBUG:teuthology.parallel:result is None 2024-04-22T01:57:20.909 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-22T01:57:20.909 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:57:21.058 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:21.055+0000 7f5af86ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:21.059 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:21.060+0000 7f5af86ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:21.243 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:57:21.243 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-04-22T01:57:21.256 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-04-22T01:57:20.842437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80924,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034596,"statfs":{"total":289910292480,"available":289827426304,"internally_reserved":0,"allocated":438272,"data_stored":1254537,"data_compressed":59920,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.957305"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T01:57:14.886124+0000","last_change":"2024-04-22T01:57:13.955407+0000","last_active":"2024-04-22T01:57:14.886124+0000","last_peered":"2024-04-22T01:57:14.886124+0000","last_clean":"2024-04-22T01:57:14.886124+0000","last_became_active":"2024-04-22T01:57:13.949422+0000","last_became_peered":"2024-04-22T01:57:13.949422+0000","last_unstale":"2024-04-22T01:57:14.886124+0000","last_undegraded":"2024-04-22T01:57:14.886124+0000","last_fullsized":"2024-04-22T01:57:14.886124+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_clean_scrub_stamp":"2024-04-22T01:57:12.853173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T01:57:14.888987+0000","last_change":"2024-04-22T01:57:14.012053+0000","last_active":"2024-04-22T01:57:14.888987+0000","last_peered":"2024-04-22T01:57:14.888987+0000","last_clean":"2024-04-22T01:57:14.888987+0000","last_became_active":"2024-04-22T01:57:14.010096+0000","last_became_peered":"2024-04-22T01:57:14.010096+0000","last_unstale":"2024-04-22T01:57:14.888987+0000","last_undegraded":"2024-04-22T01:57:14.888987+0000","last_fullsized":"2024-04-22T01:57:14.888987+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_clean_scrub_stamp":"2024-04-22T01:57:12.853173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T01:57:14.885822+0000","last_change":"2024-04-22T01:57:13.967419+0000","last_active":"2024-04-22T01:57:14.885822+0000","last_peered":"2024-04-22T01:57:14.885822+0000","last_clean":"2024-04-22T01:57:14.885822+0000","last_became_active":"2024-04-22T01:57:13.957553+0000","last_became_peered":"2024-04-22T01:57:13.957553+0000","last_unstale":"2024-04-22T01:57:14.885822+0000","last_undegraded":"2024-04-22T01:57:14.885822+0000","last_fullsized":"2024-04-22T01:57:14.885822+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_clean_scrub_stamp":"2024-04-22T01:57:12.853173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T01:57:14.887818+0000","last_change":"2024-04-22T01:57:13.981389+0000","last_active":"2024-04-22T01:57:14.887818+0000","last_peered":"2024-04-22T01:57:14.887818+0000","last_clean":"2024-04-22T01:57:14.887818+0000","last_became_active":"2024-04-22T01:57:13.979299+0000","last_became_peered":"2024-04-22T01:57:13.979299+0000","last_unstale":"2024-04-22T01:57:14.887818+0000","last_undegraded":"2024-04-22T01:57:14.887818+0000","last_fullsized":"2024-04-22T01:57:14.887818+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_clean_scrub_stamp":"2024-04-22T01:57:12.853173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T01:57:13.985962+0000","last_change":"2024-04-22T01:57:13.986167+0000","last_active":"2024-04-22T01:57:13.985962+0000","last_peered":"2024-04-22T01:57:13.985962+0000","last_clean":"2024-04-22T01:57:12.853173+0000","last_became_active":"2024-04-22T01:57:13.984104+0000","last_became_peered":"2024-04-22T01:57:13.984104+0000","last_unstale":"2024-04-22T01:57:13.985962+0000","last_undegraded":"2024-04-22T01:57:13.985962+0000","last_fullsized":"2024-04-22T01:57:13.985962+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_clean_scrub_stamp":"2024-04-22T01:57:12.853173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T01:57:13.966740+0000","last_change":"2024-04-22T01:57:13.966927+0000","last_active":"2024-04-22T01:57:13.966740+0000","last_peered":"2024-04-22T01:57:13.966740+0000","last_clean":"2024-04-22T01:57:12.853173+0000","last_became_active":"2024-04-22T01:57:13.964656+0000","last_became_peered":"2024-04-22T01:57:13.964656+0000","last_unstale":"2024-04-22T01:57:13.966740+0000","last_undegraded":"2024-04-22T01:57:13.966740+0000","last_fullsized":"2024-04-22T01:57:13.966740+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_clean_scrub_stamp":"2024-04-22T01:57:12.853173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T01:57:13.957497+0000","last_change":"2024-04-22T01:57:13.957746+0000","last_active":"2024-04-22T01:57:13.957497+0000","last_peered":"2024-04-22T01:57:13.957497+0000","last_clean":"2024-04-22T01:57:12.853173+0000","last_became_active":"2024-04-22T01:57:13.955211+0000","last_became_peered":"2024-04-22T01:57:13.955211+0000","last_unstale":"2024-04-22T01:57:13.957497+0000","last_undegraded":"2024-04-22T01:57:13.957497+0000","last_fullsized":"2024-04-22T01:57:13.957497+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_clean_scrub_stamp":"2024-04-22T01:57:12.853173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T01:57:14.888298+0000","last_change":"2024-04-22T01:57:13.931818+0000","last_active":"2024-04-22T01:57:14.888298+0000","last_peered":"2024-04-22T01:57:14.888298+0000","last_clean":"2024-04-22T01:57:14.888298+0000","last_became_active":"2024-04-22T01:57:13.927939+0000","last_became_peered":"2024-04-22T01:57:13.927939+0000","last_unstale":"2024-04-22T01:57:14.888298+0000","last_undegraded":"2024-04-22T01:57:14.888298+0000","last_fullsized":"2024-04-22T01:57:14.888298+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_clean_scrub_stamp":"2024-04-22T01:57:12.853173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T01:57:14.885401+0000","last_change":"2024-04-22T01:57:12.921379+0000","last_active":"2024-04-22T01:57:14.885401+0000","last_peered":"2024-04-22T01:57:14.885401+0000","last_clean":"2024-04-22T01:57:14.885401+0000","last_became_active":"2024-04-22T01:57:12.918029+0000","last_became_peered":"2024-04-22T01:57:12.918029+0000","last_unstale":"2024-04-22T01:57:14.885401+0000","last_undegraded":"2024-04-22T01:57:14.885401+0000","last_fullsized":"2024-04-22T01:57:14.885401+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:11.852568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:11.852568+0000","last_clean_scrub_stamp":"2024-04-22T01:57:11.852568+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":23636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":616897,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":614004,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:57:21.256 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-22T01:57:21.363 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:21.360+0000 7f951999d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:21.365 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:21.366+0000 7f951999d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:21.546 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:57:21.547 INFO:teuthology.orchestra.run.smithi152.stderr:dumped all 2024-04-22T01:57:21.559 INFO:teuthology.orchestra.run.smithi152.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-04-22T01:57:20.842437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":3,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":283115520,"kb_used":80924,"kb_used_data":428,"kb_used_omap":0,"kb_used_meta":80448,"kb_avail":283034596,"statfs":{"total":289910292480,"available":289827426304,"internally_reserved":0,"allocated":438272,"data_stored":1254537,"data_compressed":59920,"data_compressed_allocated":81920,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":82378752},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.957305"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T01:57:14.886124+0000","last_change":"2024-04-22T01:57:13.955407+0000","last_active":"2024-04-22T01:57:14.886124+0000","last_peered":"2024-04-22T01:57:14.886124+0000","last_clean":"2024-04-22T01:57:14.886124+0000","last_became_active":"2024-04-22T01:57:13.949422+0000","last_became_peered":"2024-04-22T01:57:13.949422+0000","last_unstale":"2024-04-22T01:57:14.886124+0000","last_undegraded":"2024-04-22T01:57:14.886124+0000","last_fullsized":"2024-04-22T01:57:14.886124+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_clean_scrub_stamp":"2024-04-22T01:57:12.853173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T01:57:14.888987+0000","last_change":"2024-04-22T01:57:14.012053+0000","last_active":"2024-04-22T01:57:14.888987+0000","last_peered":"2024-04-22T01:57:14.888987+0000","last_clean":"2024-04-22T01:57:14.888987+0000","last_became_active":"2024-04-22T01:57:14.010096+0000","last_became_peered":"2024-04-22T01:57:14.010096+0000","last_unstale":"2024-04-22T01:57:14.888987+0000","last_undegraded":"2024-04-22T01:57:14.888987+0000","last_fullsized":"2024-04-22T01:57:14.888987+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_clean_scrub_stamp":"2024-04-22T01:57:12.853173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T01:57:14.885822+0000","last_change":"2024-04-22T01:57:13.967419+0000","last_active":"2024-04-22T01:57:14.885822+0000","last_peered":"2024-04-22T01:57:14.885822+0000","last_clean":"2024-04-22T01:57:14.885822+0000","last_became_active":"2024-04-22T01:57:13.957553+0000","last_became_peered":"2024-04-22T01:57:13.957553+0000","last_unstale":"2024-04-22T01:57:14.885822+0000","last_undegraded":"2024-04-22T01:57:14.885822+0000","last_fullsized":"2024-04-22T01:57:14.885822+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_clean_scrub_stamp":"2024-04-22T01:57:12.853173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T01:57:14.887818+0000","last_change":"2024-04-22T01:57:13.981389+0000","last_active":"2024-04-22T01:57:14.887818+0000","last_peered":"2024-04-22T01:57:14.887818+0000","last_clean":"2024-04-22T01:57:14.887818+0000","last_became_active":"2024-04-22T01:57:13.979299+0000","last_became_peered":"2024-04-22T01:57:13.979299+0000","last_unstale":"2024-04-22T01:57:14.887818+0000","last_undegraded":"2024-04-22T01:57:14.887818+0000","last_fullsized":"2024-04-22T01:57:14.887818+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_clean_scrub_stamp":"2024-04-22T01:57:12.853173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T01:57:13.985962+0000","last_change":"2024-04-22T01:57:13.986167+0000","last_active":"2024-04-22T01:57:13.985962+0000","last_peered":"2024-04-22T01:57:13.985962+0000","last_clean":"2024-04-22T01:57:12.853173+0000","last_became_active":"2024-04-22T01:57:13.984104+0000","last_became_peered":"2024-04-22T01:57:13.984104+0000","last_unstale":"2024-04-22T01:57:13.985962+0000","last_undegraded":"2024-04-22T01:57:13.985962+0000","last_fullsized":"2024-04-22T01:57:13.985962+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_clean_scrub_stamp":"2024-04-22T01:57:12.853173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T01:57:13.966740+0000","last_change":"2024-04-22T01:57:13.966927+0000","last_active":"2024-04-22T01:57:13.966740+0000","last_peered":"2024-04-22T01:57:13.966740+0000","last_clean":"2024-04-22T01:57:12.853173+0000","last_became_active":"2024-04-22T01:57:13.964656+0000","last_became_peered":"2024-04-22T01:57:13.964656+0000","last_unstale":"2024-04-22T01:57:13.966740+0000","last_undegraded":"2024-04-22T01:57:13.966740+0000","last_fullsized":"2024-04-22T01:57:13.966740+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_clean_scrub_stamp":"2024-04-22T01:57:12.853173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":8,"reported_epoch":12,"state":"active+clean","last_fresh":"2024-04-22T01:57:13.957497+0000","last_change":"2024-04-22T01:57:13.957746+0000","last_active":"2024-04-22T01:57:13.957497+0000","last_peered":"2024-04-22T01:57:13.957497+0000","last_clean":"2024-04-22T01:57:12.853173+0000","last_became_active":"2024-04-22T01:57:13.955211+0000","last_became_peered":"2024-04-22T01:57:13.955211+0000","last_unstale":"2024-04-22T01:57:13.957497+0000","last_undegraded":"2024-04-22T01:57:13.957497+0000","last_fullsized":"2024-04-22T01:57:13.957497+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_clean_scrub_stamp":"2024-04-22T01:57:12.853173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T01:57:14.888298+0000","last_change":"2024-04-22T01:57:13.931818+0000","last_active":"2024-04-22T01:57:14.888298+0000","last_peered":"2024-04-22T01:57:14.888298+0000","last_clean":"2024-04-22T01:57:14.888298+0000","last_became_active":"2024-04-22T01:57:13.927939+0000","last_became_peered":"2024-04-22T01:57:13.927939+0000","last_unstale":"2024-04-22T01:57:14.888298+0000","last_undegraded":"2024-04-22T01:57:14.888298+0000","last_fullsized":"2024-04-22T01:57:14.888298+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:12.853173+0000","last_clean_scrub_stamp":"2024-04-22T01:57:12.853173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2024-04-22T01:57:14.885401+0000","last_change":"2024-04-22T01:57:12.921379+0000","last_active":"2024-04-22T01:57:14.885401+0000","last_peered":"2024-04-22T01:57:14.885401+0000","last_clean":"2024-04-22T01:57:14.885401+0000","last_became_active":"2024-04-22T01:57:12.918029+0000","last_became_peered":"2024-04-22T01:57:12.918029+0000","last_unstale":"2024-04-22T01:57:14.885401+0000","last_undegraded":"2024-04-22T01:57:14.885401+0000","last_fullsized":"2024-04-22T01:57:14.885401+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T01:57:11.852568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T01:57:11.852568+0000","last_clean_scrub_stamp":"2024-04-22T01:57:11.852568+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":103,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":2,"up_from":9,"seq":38654705669,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26924,"kb_used_data":100,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344916,"statfs":{"total":96636764160,"available":96609193984,"internally_reserved":0,"allocated":102400,"data_stored":23636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27004,"kb_used_data":168,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344836,"statfs":{"total":96636764160,"available":96609112064,"internally_reserved":0,"allocated":172032,"data_stored":616897,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":26996,"kb_used_data":160,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94344844,"statfs":{"total":96636764160,"available":96609120256,"internally_reserved":0,"allocated":163840,"data_stored":614004,"data_compressed":29960,"data_compressed_allocated":40960,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[]}} 2024-04-22T01:57:21.560 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-22T01:57:21.560 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-22T01:57:21.560 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-22T01:57:21.560 DEBUG:teuthology.orchestra.run.smithi152:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-22T01:57:21.667 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:21.665+0000 7f5bcb954640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:21.669 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:57:21.669+0000 7f5bcb954640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:57:21.894 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:57:21.894 INFO:teuthology.orchestra.run.smithi152.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-22T01:57:21.907 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-22T01:57:21.907 INFO:teuthology.run_tasks:Running task cbt... 2024-04-22T01:57:21.926 INFO:tasks.cbt:cbt configuration is {'cluster': {'user': 'ubuntu', 'head': 'ubuntu@smithi152.front.sepia.ceph.com', 'osds': ['ubuntu@smithi152.front.sepia.ceph.com'], 'mons': ['ubuntu@smithi152.front.sepia.ceph.com'], 'clients': ['ubuntu@smithi152.front.sepia.ceph.com'], 'rgws': {'ubuntu@smithi152.front.sepia.ceph.com': None}, 'osds_per_node': 3, 'rebuild_every_test': False, 'use_existing': True, 'is_teuthology': True, 'iterations': 1, 'tmp_dir': '/tmp/cbt', 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 'pid_dir': '/var/run/ceph'}, 'benchmarks': {'librbdfio': {'iodepth': [32], 'log_avg_msec': 100, 'mode': ['randrw'], 'norandommap': True, 'op_size': [4194304], 'osd_ra': [4096], 'pool_profile': 'rbd', 'procs_per_volume': [1], 'time': 60, 'vol_size': 4096, 'volumes_per_client': [2], 'cmd_path': '/home/ubuntu/cephtest/fio/fio'}}, 'client_endpoints': None, 'monitoring_profiles': {'perf': {'args': 'stat -p {pid} -o {perf_dir}/perf_stat.{pid}', 'nodes': ['osds'], 'perf_cmd': 'perf', 'pid_glob': 'crimson-osd.*.pid'}}} 2024-04-22T01:57:21.927 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/cbt 2024-04-22T01:57:21.944 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2024-04-22T01:57:21.945 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-04-22T01:57:21.997 DEBUG:teuthology.orchestra.run.smithi152:> git clone --depth 1 -b master https://github.com/ceph/cbt.git /home/ubuntu/cephtest/cbt 2024-04-22T01:57:22.052 INFO:teuthology.orchestra.run.smithi152.stderr:Cloning into '/home/ubuntu/cephtest/cbt'... 2024-04-22T01:57:22.474 INFO:tasks.cbt:Installing collectl 2024-04-22T01:57:22.474 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir collectl && cd collectl && sudo wget https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download -O collectl.tar.gz && sudo tar -xvf collectl.tar.gz && cd collectl-4.3.1 && sudo ./INSTALL 2024-04-22T01:57:22.558 INFO:teuthology.orchestra.run.smithi152.stderr:--2024-04-22 01:57:22-- https://sourceforge.net/projects/collectl/files/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz/download 2024-04-22T01:57:22.688 INFO:teuthology.orchestra.run.smithi152.stderr:Resolving sourceforge.net (sourceforge.net)... 172.64.150.145, 104.18.37.111, 2606:4700:4400::6812:256f, ... 2024-04-22T01:57:22.698 INFO:teuthology.orchestra.run.smithi152.stderr:Connecting to sourceforge.net (sourceforge.net)|172.64.150.145|:443... connected. 2024-04-22T01:57:22.917 INFO:teuthology.orchestra.run.smithi152.stderr:HTTP request sent, awaiting response... 302 Found 2024-04-22T01:57:22.917 INFO:teuthology.orchestra.run.smithi152.stderr:Location: https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABmJcQCZHxf5Oqoj2Lz5M4uKxBVK5RKTQr4Wy9rHxuju6AQN0G4TTprhXKhc-i6yGzaTVCpTVNUEupUxyK17_1eQBPmxQ%3D%3D&use_mirror=pilotfiber&r= [following] 2024-04-22T01:57:22.918 INFO:teuthology.orchestra.run.smithi152.stderr:--2024-04-22 01:57:22-- https://downloads.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz?ts=gAAAAABmJcQCZHxf5Oqoj2Lz5M4uKxBVK5RKTQr4Wy9rHxuju6AQN0G4TTprhXKhc-i6yGzaTVCpTVNUEupUxyK17_1eQBPmxQ%3D%3D&use_mirror=pilotfiber&r= 2024-04-22T01:57:22.929 INFO:teuthology.orchestra.run.smithi152.stderr:Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 204.68.111.105 2024-04-22T01:57:22.997 INFO:teuthology.orchestra.run.smithi152.stderr:Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|204.68.111.105|:443... connected. 2024-04-22T01:57:23.258 INFO:teuthology.orchestra.run.smithi152.stderr:HTTP request sent, awaiting response... 302 Found 2024-04-22T01:57:23.258 INFO:teuthology.orchestra.run.smithi152.stderr:Location: https://pilotfiber.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz [following] 2024-04-22T01:57:23.258 INFO:teuthology.orchestra.run.smithi152.stderr:--2024-04-22 01:57:23-- https://pilotfiber.dl.sourceforge.net/project/collectl/collectl/collectl-4.3.1/collectl-4.3.1.src.tar.gz 2024-04-22T01:57:23.270 INFO:teuthology.orchestra.run.smithi152.stderr:Resolving pilotfiber.dl.sourceforge.net (pilotfiber.dl.sourceforge.net)... 216.158.133.250 2024-04-22T01:57:23.287 INFO:teuthology.orchestra.run.smithi152.stderr:Connecting to pilotfiber.dl.sourceforge.net (pilotfiber.dl.sourceforge.net)|216.158.133.250|:443... connected. 2024-04-22T01:57:23.434 INFO:teuthology.orchestra.run.smithi152.stderr:HTTP request sent, awaiting response... 200 OK 2024-04-22T01:57:23.434 INFO:teuthology.orchestra.run.smithi152.stderr:Length: 620693 (606K) [application/x-gzip] 2024-04-22T01:57:23.434 INFO:teuthology.orchestra.run.smithi152.stderr:Saving to: ‘collectl.tar.gz’ 2024-04-22T01:57:23.434 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-04-22T01:57:23.451 INFO:teuthology.orchestra.run.smithi152.stderr: 0K .......... .......... .......... .......... .......... 8% 2.86M 0s 2024-04-22T01:57:23.468 INFO:teuthology.orchestra.run.smithi152.stderr: 50K .......... .......... .......... .......... .......... 16% 2.71M 0s 2024-04-22T01:57:23.486 INFO:teuthology.orchestra.run.smithi152.stderr: 100K .......... .......... .......... .......... .......... 24% 3.09M 0s 2024-04-22T01:57:23.486 INFO:teuthology.orchestra.run.smithi152.stderr: 150K .......... .......... .......... .......... .......... 32% 112M 0s 2024-04-22T01:57:23.502 INFO:teuthology.orchestra.run.smithi152.stderr: 200K .......... .......... .......... .......... .......... 41% 2.97M 0s 2024-04-22T01:57:23.545 INFO:teuthology.orchestra.run.smithi152.stderr: 250K .......... .......... .......... .......... .......... 49% 1.16M 0s 2024-04-22T01:57:23.545 INFO:teuthology.orchestra.run.smithi152.stderr: 300K .......... .......... .......... .......... .......... 57% 133M 0s 2024-04-22T01:57:23.545 INFO:teuthology.orchestra.run.smithi152.stderr: 350K .......... .......... .......... .......... .......... 65% 87.6M 0s 2024-04-22T01:57:23.549 INFO:teuthology.orchestra.run.smithi152.stderr: 400K .......... .......... .......... .......... .......... 74% 18.6M 0s 2024-04-22T01:57:23.550 INFO:teuthology.orchestra.run.smithi152.stderr: 450K .......... .......... .......... .......... .......... 82% 53.5M 0s 2024-04-22T01:57:23.550 INFO:teuthology.orchestra.run.smithi152.stderr: 500K .......... .......... .......... .......... .......... 90% 118M 0s 2024-04-22T01:57:23.561 INFO:teuthology.orchestra.run.smithi152.stderr: 550K .......... .......... .......... .......... .......... 98% 4.04M 0s 2024-04-22T01:57:23.561 INFO:teuthology.orchestra.run.smithi152.stderr: 600K ...... 100% 11.4T=0.1s 2024-04-22T01:57:23.561 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-04-22T01:57:23.561 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22 01:57:23 (4.67 MB/s) - ‘collectl.tar.gz’ saved [620693/620693] 2024-04-22T01:57:23.561 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-04-22T01:57:23.593 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/ 2024-04-22T01:57:23.593 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/collectl.conf 2024-04-22T01:57:23.594 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/ 2024-04-22T01:57:23.594 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/SlowProc.html 2024-04-22T01:57:23.594 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/BuddyInfo.html 2024-04-22T01:57:23.594 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/WhySummary.html 2024-04-22T01:57:23.594 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Socket.html 2024-04-22T01:57:23.594 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/colmux.html 2024-04-22T01:57:23.594 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Tutorial.html 2024-04-22T01:57:23.594 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Gexpr.html 2024-04-22T01:57:23.594 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Import.html 2024-04-22T01:57:23.594 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/style.css 2024-04-22T01:57:23.594 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Ganglia3.jpg 2024-04-22T01:57:23.594 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Ganglia2.jpg 2024-04-22T01:57:23.594 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/FileNaming.html 2024-04-22T01:57:23.595 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/TheMath.html 2024-04-22T01:57:23.595 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Environmental.html 2024-04-22T01:57:23.595 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Process.html 2024-04-22T01:57:23.595 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/OutputFiles.html 2024-04-22T01:57:23.595 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/HiResTime.html 2024-04-22T01:57:23.595 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/OutputFormats.html 2024-04-22T01:57:23.595 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/slub.jpg 2024-04-22T01:57:23.595 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Lustre.html 2024-04-22T01:57:23.595 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Documentation.html 2024-04-22T01:57:23.595 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Inodes.html 2024-04-22T01:57:23.595 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Graphite.html 2024-04-22T01:57:23.595 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/InputFiles.html 2024-04-22T01:57:23.596 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Export.html 2024-04-22T01:57:23.596 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Interrupts.html 2024-04-22T01:57:23.596 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Startup.html 2024-04-22T01:57:23.596 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Performance.html 2024-04-22T01:57:23.596 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Messages.html 2024-04-22T01:57:23.596 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Logging.html 2024-04-22T01:57:23.596 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Data-detail.html 2024-04-22T01:57:23.596 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/CPUs.html 2024-04-22T01:57:23.596 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Matrix.html 2024-04-22T01:57:23.596 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Data-brief.html 2024-04-22T01:57:23.596 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/SlabInfo.html 2024-04-22T01:57:23.596 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Misc.html 2024-04-22T01:57:23.596 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Examples.html 2024-04-22T01:57:23.596 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/OpenStack.html 2024-04-22T01:57:23.597 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Plotfiles.html 2024-04-22T01:57:23.597 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/ColmuxCPU.jpg 2024-04-22T01:57:23.597 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Tutorial-Lustre.html 2024-04-22T01:57:23.597 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Data-verbose.html 2024-04-22T01:57:23.597 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Data.html 2024-04-22T01:57:23.597 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Exceptions.html 2024-04-22T01:57:23.597 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/RunningAsAService.html 2024-04-22T01:57:23.597 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/FAQ-collectl.html 2024-04-22T01:57:23.597 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Ganglia.jpg 2024-04-22T01:57:23.597 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/OperationalModes.html 2024-04-22T01:57:23.597 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Playback.html 2024-04-22T01:57:23.597 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Hello.html 2024-04-22T01:57:23.597 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Sockets.html 2024-04-22T01:57:23.598 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Network.html 2024-04-22T01:57:23.598 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Infiniband.html 2024-04-22T01:57:23.598 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Memory.html 2024-04-22T01:57:23.598 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/ColmuxLustre.jpg 2024-04-22T01:57:23.598 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/Disks.html 2024-04-22T01:57:23.598 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/docs/NfsInfo.html 2024-04-22T01:57:23.598 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/service/ 2024-04-22T01:57:23.598 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/service/collectl.service 2024-04-22T01:57:23.598 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/initd/ 2024-04-22T01:57:23.598 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/initd/collectl-suse 2024-04-22T01:57:23.598 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/initd/collectl-debian 2024-04-22T01:57:23.599 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/initd/collectl-generic 2024-04-22T01:57:23.599 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/initd/collectl 2024-04-22T01:57:23.599 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/misc.ph 2024-04-22T01:57:23.599 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/gexpr.ph 2024-04-22T01:57:23.599 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/statsd.ph 2024-04-22T01:57:23.599 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/envrules.std 2024-04-22T01:57:23.599 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/hello.ph 2024-04-22T01:57:23.599 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/vmsum.ph 2024-04-22T01:57:23.599 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/proctree.ph 2024-04-22T01:57:23.599 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/README-WINDOWS 2024-04-22T01:57:23.599 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/formatit.ph 2024-04-22T01:57:23.599 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/colmux 2024-04-22T01:57:23.599 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/INSTALL 2024-04-22T01:57:23.600 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/graphite.ph 2024-04-22T01:57:23.600 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/RELEASE-collectl 2024-04-22T01:57:23.600 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/vmstat.ph 2024-04-22T01:57:23.600 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/README 2024-04-22T01:57:23.600 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/man1/ 2024-04-22T01:57:23.600 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/man1/collectl.1 2024-04-22T01:57:23.600 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/man1/colmux.1 2024-04-22T01:57:23.600 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/lexpr.ph 2024-04-22T01:57:23.600 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/client.pl 2024-04-22T01:57:23.600 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/collectl 2024-04-22T01:57:23.601 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/COPYING 2024-04-22T01:57:23.602 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/vnet.ph 2024-04-22T01:57:23.602 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/GPL 2024-04-22T01:57:23.602 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/UNINSTALL 2024-04-22T01:57:23.602 INFO:teuthology.orchestra.run.smithi152.stdout:collectl-4.3.1/ARTISTIC 2024-04-22T01:57:23.643 INFO:teuthology.orchestra.run.smithi152.stderr:./INSTALL: line 90: chkconfig: command not found 2024-04-22T01:57:23.645 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-yaml python3-lxml librbd-devel pdsh pdsh-rcmd-ssh perf 2024-04-22T01:57:24.213 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:03:21 ago on Mon 22 Apr 2024 01:54:03 AM UTC. 2024-04-22T01:57:24.293 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-pyyaml-5.4.1-6.el9.x86_64 is already installed. 2024-04-22T01:57:24.295 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-lxml-4.6.5-3.el9.x86_64 is already installed. 2024-04-22T01:57:24.341 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2024-04-22T01:57:24.341 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:57:24.341 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2024-04-22T01:57:24.341 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:57:24.341 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2024-04-22T01:57:24.342 INFO:teuthology.orchestra.run.smithi152.stdout: librbd-devel x86_64 2:19.0.0-3159.g676947e1.el9 ceph 23 k 2024-04-22T01:57:24.342 INFO:teuthology.orchestra.run.smithi152.stdout: pdsh x86_64 2.34-7.el9 epel 118 k 2024-04-22T01:57:24.342 INFO:teuthology.orchestra.run.smithi152.stdout: pdsh-rcmd-ssh x86_64 2.34-7.el9 epel 13 k 2024-04-22T01:57:24.342 INFO:teuthology.orchestra.run.smithi152.stdout: perf x86_64 5.14.0-437.el9 appstream 2.2 M 2024-04-22T01:57:24.342 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2024-04-22T01:57:24.342 INFO:teuthology.orchestra.run.smithi152.stdout: libradospp-devel x86_64 2:19.0.0-3159.g676947e1.el9 ceph 30 k 2024-04-22T01:57:24.342 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:57:24.342 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2024-04-22T01:57:24.342 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2024-04-22T01:57:24.342 INFO:teuthology.orchestra.run.smithi152.stdout:Install 5 Packages 2024-04-22T01:57:24.342 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:57:24.343 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 2.4 M 2024-04-22T01:57:24.343 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 11 M 2024-04-22T01:57:24.343 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2024-04-22T01:57:25.138 INFO:teuthology.orchestra.run.smithi152.stdout:(1/5): perf-5.14.0-437.el9.x86_64.rpm 6.1 MB/s | 2.2 MB 00:00 2024-04-22T01:57:25.155 INFO:teuthology.orchestra.run.smithi152.stdout:(2/5): libradospp-devel-19.0.0-3159.g676947e1.e 80 kB/s | 30 kB 00:00 2024-04-22T01:57:25.338 INFO:teuthology.orchestra.run.smithi152.stdout:(3/5): librbd-devel-19.0.0-3159.g676947e1.el9.x 41 kB/s | 23 kB 00:00 2024-04-22T01:57:25.355 INFO:teuthology.orchestra.run.smithi152.stdout:(4/5): pdsh-rcmd-ssh-2.34-7.el9.x86_64.rpm 65 kB/s | 13 kB 00:00 2024-04-22T01:57:25.414 INFO:teuthology.orchestra.run.smithi152.stdout:(5/5): pdsh-2.34-7.el9.x86_64.rpm 431 kB/s | 118 kB 00:00 2024-04-22T01:57:25.418 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2024-04-22T01:57:25.418 INFO:teuthology.orchestra.run.smithi152.stdout:Total 2.2 MB/s | 2.4 MB 00:01 2024-04-22T01:57:25.448 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2024-04-22T01:57:25.469 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2024-04-22T01:57:25.469 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2024-04-22T01:57:25.576 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2024-04-22T01:57:25.577 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2024-04-22T01:57:25.914 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2024-04-22T01:57:25.983 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : pdsh-rcmd-ssh-2.34-7.el9.x86_64 1/5 2024-04-22T01:57:26.013 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : pdsh-2.34-7.el9.x86_64 2/5 2024-04-22T01:57:26.044 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libradospp-devel-2:19.0.0-3159.g676947e1.el9.x86_64 3/5 2024-04-22T01:57:26.358 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librbd-devel-2:19.0.0-3159.g676947e1.el9.x86_64 4/5 2024-04-22T01:57:26.449 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : perf-5.14.0-437.el9.x86_64 5/5 2024-04-22T01:57:27.364 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: perf-5.14.0-437.el9.x86_64 5/5 2024-04-22T01:57:27.364 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libradospp-devel-2:19.0.0-3159.g676947e1.el9.x86_64 1/5 2024-04-22T01:57:27.364 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd-devel-2:19.0.0-3159.g676947e1.el9.x86_64 2/5 2024-04-22T01:57:27.365 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : perf-5.14.0-437.el9.x86_64 3/5 2024-04-22T01:57:27.365 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : pdsh-2.34-7.el9.x86_64 4/5 2024-04-22T01:57:27.647 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : pdsh-rcmd-ssh-2.34-7.el9.x86_64 5/5 2024-04-22T01:57:27.647 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:57:27.647 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2024-04-22T01:57:27.647 INFO:teuthology.orchestra.run.smithi152.stdout: libradospp-devel-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:57:27.647 INFO:teuthology.orchestra.run.smithi152.stdout: librbd-devel-2:19.0.0-3159.g676947e1.el9.x86_64 2024-04-22T01:57:27.647 INFO:teuthology.orchestra.run.smithi152.stdout: pdsh-2.34-7.el9.x86_64 2024-04-22T01:57:27.648 INFO:teuthology.orchestra.run.smithi152.stdout: pdsh-rcmd-ssh-2.34-7.el9.x86_64 2024-04-22T01:57:27.648 INFO:teuthology.orchestra.run.smithi152.stdout: perf-5.14.0-437.el9.x86_64 2024-04-22T01:57:27.648 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:57:27.648 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2024-04-22T01:57:28.003 INFO:tasks.cbt:benchmark: librbdfio 2024-04-22T01:57:28.004 DEBUG:teuthology.orchestra.run.smithi152:> git clone -b master https://github.com/axboe/fio.git /home/ubuntu/cephtest/fio 2024-04-22T01:57:28.018 INFO:teuthology.orchestra.run.smithi152.stderr:Cloning into '/home/ubuntu/cephtest/fio'... 2024-04-22T01:57:30.223 DEBUG:teuthology.orchestra.run.smithi152:> cd /home/ubuntu/cephtest/fio && ./configure && make 2024-04-22T01:57:30.755 INFO:teuthology.orchestra.run.smithi152.stdout:Operating system Linux 2024-04-22T01:57:30.755 INFO:teuthology.orchestra.run.smithi152.stdout:CPU x86_64 2024-04-22T01:57:30.755 INFO:teuthology.orchestra.run.smithi152.stdout:Big endian no 2024-04-22T01:57:30.755 INFO:teuthology.orchestra.run.smithi152.stdout:Compiler gcc 2024-04-22T01:57:30.755 INFO:teuthology.orchestra.run.smithi152.stdout:Cross compile no 2024-04-22T01:57:30.755 INFO:teuthology.orchestra.run.smithi152.stdout: 2024-04-22T01:57:30.755 INFO:teuthology.orchestra.run.smithi152.stdout:Static build no 2024-04-22T01:57:30.842 INFO:teuthology.orchestra.run.smithi152.stdout:Wordsize 64 2024-04-22T01:57:30.849 INFO:teuthology.orchestra.run.smithi152.stdout:zlib no 2024-04-22T01:57:30.889 INFO:teuthology.orchestra.run.smithi152.stdout:fcntl(F_FULLFSYNC) no 2024-04-22T01:57:30.903 INFO:teuthology.orchestra.run.smithi152.stdout:Linux AIO support no 2024-04-22T01:57:30.904 INFO:teuthology.orchestra.run.smithi152.stdout:Linux AIO support rw flags no 2024-04-22T01:57:30.956 INFO:teuthology.orchestra.run.smithi152.stdout:POSIX AIO support yes 2024-04-22T01:57:30.957 INFO:teuthology.orchestra.run.smithi152.stdout:POSIX AIO support needs -lrt no 2024-04-22T01:57:30.987 INFO:teuthology.orchestra.run.smithi152.stdout:POSIX AIO fsync yes 2024-04-22T01:57:31.017 INFO:teuthology.orchestra.run.smithi152.stdout:POSIX pshared support yes 2024-04-22T01:57:31.054 INFO:teuthology.orchestra.run.smithi152.stdout:pthread_condattr_setclock() yes 2024-04-22T01:57:31.092 INFO:teuthology.orchestra.run.smithi152.stdout:pthread_sigmask() yes 2024-04-22T01:57:31.128 INFO:teuthology.orchestra.run.smithi152.stdout:pthread_getaffinity_np() yes 2024-04-22T01:57:31.137 INFO:teuthology.orchestra.run.smithi152.stdout:Solaris AIO support no 2024-04-22T01:57:31.189 INFO:teuthology.orchestra.run.smithi152.stdout:__sync_fetch_and_add yes 2024-04-22T01:57:31.238 INFO:teuthology.orchestra.run.smithi152.stdout:__sync_synchronize yes 2024-04-22T01:57:31.266 INFO:teuthology.orchestra.run.smithi152.stdout:__sync_val_compare_and_swap yes 2024-04-22T01:57:31.274 INFO:teuthology.orchestra.run.smithi152.stdout:libverbs no 2024-04-22T01:57:31.289 INFO:teuthology.orchestra.run.smithi152.stdout:rdmacm no 2024-04-22T01:57:31.297 INFO:teuthology.orchestra.run.smithi152.stdout:librpma no 2024-04-22T01:57:31.308 INFO:teuthology.orchestra.run.smithi152.stdout:libprotobuf_c no 2024-04-22T01:57:31.338 INFO:teuthology.orchestra.run.smithi152.stdout:asprintf() yes 2024-04-22T01:57:31.367 INFO:teuthology.orchestra.run.smithi152.stdout:vasprintf() yes 2024-04-22T01:57:31.399 INFO:teuthology.orchestra.run.smithi152.stdout:Linux fallocate yes 2024-04-22T01:57:31.430 INFO:teuthology.orchestra.run.smithi152.stdout:POSIX fadvise yes 2024-04-22T01:57:31.461 INFO:teuthology.orchestra.run.smithi152.stdout:POSIX fallocate yes 2024-04-22T01:57:31.493 INFO:teuthology.orchestra.run.smithi152.stdout:sched_setaffinity(3 arg) yes 2024-04-22T01:57:31.493 INFO:teuthology.orchestra.run.smithi152.stdout:sched_setaffinity(2 arg) no 2024-04-22T01:57:31.525 INFO:teuthology.orchestra.run.smithi152.stdout:clock_gettime yes 2024-04-22T01:57:31.557 INFO:teuthology.orchestra.run.smithi152.stdout:CLOCK_MONOTONIC yes 2024-04-22T01:57:31.588 INFO:teuthology.orchestra.run.smithi152.stdout:clockid_t yes 2024-04-22T01:57:31.620 INFO:teuthology.orchestra.run.smithi152.stdout:gettimeofday yes 2024-04-22T01:57:31.652 INFO:teuthology.orchestra.run.smithi152.stdout:fdatasync yes 2024-04-22T01:57:31.682 INFO:teuthology.orchestra.run.smithi152.stdout:pipe() yes 2024-04-22T01:57:31.713 INFO:teuthology.orchestra.run.smithi152.stdout:pipe2() yes 2024-04-22T01:57:31.743 INFO:teuthology.orchestra.run.smithi152.stdout:pread() yes 2024-04-22T01:57:31.828 INFO:teuthology.orchestra.run.smithi152.stdout:sync_file_range yes 2024-04-22T01:57:31.835 INFO:teuthology.orchestra.run.smithi152.stdout:ASharedMemory_create no 2024-04-22T01:57:31.849 INFO:teuthology.orchestra.run.smithi152.stdout:EXT4 move extent yes 2024-04-22T01:57:31.880 INFO:teuthology.orchestra.run.smithi152.stdout:Linux splice(2) yes 2024-04-22T01:57:31.888 INFO:teuthology.orchestra.run.smithi152.stdout:libnuma no 2024-04-22T01:57:31.918 INFO:teuthology.orchestra.run.smithi152.stdout:strsep yes 2024-04-22T01:57:31.948 INFO:teuthology.orchestra.run.smithi152.stdout:strcasestr yes 2024-04-22T01:57:31.963 INFO:teuthology.orchestra.run.smithi152.stdout:strlcat no 2024-04-22T01:57:31.996 INFO:teuthology.orchestra.run.smithi152.stdout:getopt_long_only() yes 2024-04-22T01:57:32.045 INFO:teuthology.orchestra.run.smithi152.stdout:inet_aton yes 2024-04-22T01:57:32.077 INFO:teuthology.orchestra.run.smithi152.stdout:socklen_t yes 2024-04-22T01:57:32.118 INFO:teuthology.orchestra.run.smithi152.stdout:__thread yes 2024-04-22T01:57:32.156 INFO:teuthology.orchestra.run.smithi152.stdout:RUSAGE_THREAD yes 2024-04-22T01:57:32.185 INFO:teuthology.orchestra.run.smithi152.stdout:SCHED_IDLE yes 2024-04-22T01:57:32.219 INFO:teuthology.orchestra.run.smithi152.stdout:TCP_NODELAY yes 2024-04-22T01:57:32.261 INFO:teuthology.orchestra.run.smithi152.stdout:vsock yes 2024-04-22T01:57:32.295 INFO:teuthology.orchestra.run.smithi152.stdout:Net engine window_size yes 2024-04-22T01:57:32.328 INFO:teuthology.orchestra.run.smithi152.stdout:TCP_MAXSEG yes 2024-04-22T01:57:32.357 INFO:teuthology.orchestra.run.smithi152.stdout:RLIMIT_MEMLOCK yes 2024-04-22T01:57:32.392 INFO:teuthology.orchestra.run.smithi152.stdout:pwritev/preadv yes 2024-04-22T01:57:32.425 INFO:teuthology.orchestra.run.smithi152.stdout:pwritev2/preadv2 yes 2024-04-22T01:57:32.467 INFO:teuthology.orchestra.run.smithi152.stdout:IPv6 helpers yes 2024-04-22T01:57:32.483 INFO:teuthology.orchestra.run.smithi152.stdout:http engine no 2024-04-22T01:57:32.625 INFO:teuthology.orchestra.run.smithi152.stdout:Rados engine yes 2024-04-22T01:57:32.781 INFO:teuthology.orchestra.run.smithi152.stdout:Rados Block Device engine yes 2024-04-22T01:57:32.937 INFO:teuthology.orchestra.run.smithi152.stdout:rbd_poll yes 2024-04-22T01:57:33.089 INFO:teuthology.orchestra.run.smithi152.stdout:rbd_invalidate_cache yes 2024-04-22T01:57:33.120 INFO:teuthology.orchestra.run.smithi152.stdout:setvbuf yes 2024-04-22T01:57:33.127 INFO:teuthology.orchestra.run.smithi152.stdout:Gluster API engine no 2024-04-22T01:57:33.143 INFO:teuthology.orchestra.run.smithi152.stdout:s390_z196_facilities no 2024-04-22T01:57:33.144 INFO:teuthology.orchestra.run.smithi152.stdout:HDFS engine no 2024-04-22T01:57:33.187 INFO:teuthology.orchestra.run.smithi152.stdout:MTD yes 2024-04-22T01:57:33.194 INFO:teuthology.orchestra.run.smithi152.stdout:libpmem no 2024-04-22T01:57:33.194 INFO:teuthology.orchestra.run.smithi152.stdout:libpmem1_5 no 2024-04-22T01:57:33.203 INFO:teuthology.orchestra.run.smithi152.stdout:libpmem2 no 2024-04-22T01:57:33.203 INFO:teuthology.orchestra.run.smithi152.stdout:PMDK dev-dax engine no 2024-04-22T01:57:33.203 INFO:teuthology.orchestra.run.smithi152.stdout:PMDK libpmem engine no 2024-04-22T01:57:33.210 INFO:teuthology.orchestra.run.smithi152.stdout:DDN's Infinite Memory Engine no 2024-04-22T01:57:33.210 INFO:teuthology.orchestra.run.smithi152.stdout:iscsi engine no 2024-04-22T01:57:33.210 INFO:teuthology.orchestra.run.smithi152.stdout:NBD engine no 2024-04-22T01:57:33.219 INFO:teuthology.orchestra.run.smithi152.stdout:DAOS File System (dfs) Engine no 2024-04-22T01:57:33.228 INFO:teuthology.orchestra.run.smithi152.stdout:NFS engine no 2024-04-22T01:57:33.228 INFO:teuthology.orchestra.run.smithi152.stdout:lex/yacc for arithmetic no 2024-04-22T01:57:33.259 INFO:teuthology.orchestra.run.smithi152.stdout:getmntent yes 2024-04-22T01:57:33.284 INFO:teuthology.orchestra.run.smithi152.stdout:getmntinfo no 2024-04-22T01:57:33.330 INFO:teuthology.orchestra.run.smithi152.stdout:Static Assert yes 2024-04-22T01:57:33.357 INFO:teuthology.orchestra.run.smithi152.stdout:bool yes 2024-04-22T01:57:33.390 INFO:teuthology.orchestra.run.smithi152.stdout:strndup yes 2024-04-22T01:57:33.398 INFO:teuthology.orchestra.run.smithi152.stdout:Valgrind headers no 2024-04-22T01:57:33.426 INFO:teuthology.orchestra.run.smithi152.stdout:Zoned block device support yes 2024-04-22T01:57:33.454 INFO:teuthology.orchestra.run.smithi152.stdout:Zoned block device capacity yes 2024-04-22T01:57:33.462 INFO:teuthology.orchestra.run.smithi152.stdout:libzbc engine no 2024-04-22T01:57:33.498 INFO:teuthology.orchestra.run.smithi152.stdout:NVMe uring command support yes 2024-04-22T01:57:33.513 INFO:teuthology.orchestra.run.smithi152.stdout:xnvme engine no 2024-04-22T01:57:33.520 INFO:teuthology.orchestra.run.smithi152.stdout:isal no 2024-04-22T01:57:33.535 INFO:teuthology.orchestra.run.smithi152.stdout:libblkio engine no 2024-04-22T01:57:33.535 INFO:teuthology.orchestra.run.smithi152.stdout:march_armv8_a_crc_crypto no 2024-04-22T01:57:33.535 INFO:teuthology.orchestra.run.smithi152.stdout:cuda no 2024-04-22T01:57:33.535 INFO:teuthology.orchestra.run.smithi152.stdout:libcufile no 2024-04-22T01:57:33.563 INFO:teuthology.orchestra.run.smithi152.stdout:Build march=native yes 2024-04-22T01:57:33.570 INFO:teuthology.orchestra.run.smithi152.stdout:CUnit no 2024-04-22T01:57:33.598 INFO:teuthology.orchestra.run.smithi152.stdout:__kernel_rwf_t yes 2024-04-22T01:57:33.626 INFO:teuthology.orchestra.run.smithi152.stdout:-Wimplicit-fallthrough=2 yes 2024-04-22T01:57:33.656 INFO:teuthology.orchestra.run.smithi152.stdout:-Wno-stringop-truncation yes 2024-04-22T01:57:33.696 INFO:teuthology.orchestra.run.smithi152.stdout:MADV_HUGEPAGE yes 2024-04-22T01:57:33.727 INFO:teuthology.orchestra.run.smithi152.stdout:gettid yes 2024-04-22T01:57:33.763 INFO:teuthology.orchestra.run.smithi152.stdout:statx(2)/libc yes 2024-04-22T01:57:33.797 INFO:teuthology.orchestra.run.smithi152.stdout:statx(2)/syscall yes 2024-04-22T01:57:33.799 INFO:teuthology.orchestra.run.smithi152.stdout:Windows PDB generation no 2024-04-22T01:57:33.829 INFO:teuthology.orchestra.run.smithi152.stdout:timerfd_create yes 2024-04-22T01:57:33.831 INFO:teuthology.orchestra.run.smithi152.stdout:Consider installing zlib1g-dev (zlib-devel) as some fio features depend on it. 2024-04-22T01:57:33.832 INFO:teuthology.orchestra.run.smithi152.stdout:Lib-based ioengines dynamic no 2024-04-22T01:57:33.896 INFO:teuthology.orchestra.run.smithi152.stdout:TCMalloc support yes 2024-04-22T01:57:33.897 INFO:teuthology.orchestra.run.smithi152.stdout:seed_buckets 4 2024-04-22T01:57:33.975 INFO:teuthology.orchestra.run.smithi152.stderr:FIO_VERSION = fio-3.37-15-gc948 2024-04-22T01:57:34.009 INFO:teuthology.orchestra.run.smithi152.stdout: CC crc/crc16.o 2024-04-22T01:57:34.087 INFO:teuthology.orchestra.run.smithi152.stdout: CC crc/crc32.o 2024-04-22T01:57:34.125 INFO:teuthology.orchestra.run.smithi152.stdout: CC crc/crc32c-arm64.o 2024-04-22T01:57:34.211 INFO:teuthology.orchestra.run.smithi152.stdout: CC crc/crc32c-intel.o 2024-04-22T01:57:34.256 INFO:teuthology.orchestra.run.smithi152.stdout: CC crc/crc32c.o 2024-04-22T01:57:34.295 INFO:teuthology.orchestra.run.smithi152.stdout: CC crc/crc64.o 2024-04-22T01:57:34.335 INFO:teuthology.orchestra.run.smithi152.stdout: CC crc/crc7.o 2024-04-22T01:57:34.368 INFO:teuthology.orchestra.run.smithi152.stdout: CC crc/crct10dif_common.o 2024-04-22T01:57:34.402 INFO:teuthology.orchestra.run.smithi152.stdout: CC crc/fnv.o 2024-04-22T01:57:34.442 INFO:teuthology.orchestra.run.smithi152.stdout: CC crc/md5.o 2024-04-22T01:57:34.545 INFO:teuthology.orchestra.run.smithi152.stdout: CC crc/murmur3.o 2024-04-22T01:57:34.587 INFO:teuthology.orchestra.run.smithi152.stdout: CC crc/sha1.o 2024-04-22T01:57:34.773 INFO:teuthology.orchestra.run.smithi152.stdout: CC crc/sha256.o 2024-04-22T01:57:35.063 INFO:teuthology.orchestra.run.smithi152.stdout: CC crc/sha3.o 2024-04-22T01:57:35.209 INFO:teuthology.orchestra.run.smithi152.stdout: CC crc/sha512.o 2024-04-22T01:57:35.325 INFO:teuthology.orchestra.run.smithi152.stdout: CC crc/test.o 2024-04-22T01:57:35.490 INFO:teuthology.orchestra.run.smithi152.stdout: CC crc/xxhash.o 2024-04-22T01:57:35.586 INFO:teuthology.orchestra.run.smithi152.stdout: CC lib/axmap.o 2024-04-22T01:57:35.711 INFO:teuthology.orchestra.run.smithi152.stdout: CC lib/bloom.o 2024-04-22T01:57:35.795 INFO:teuthology.orchestra.run.smithi152.stdout: CC lib/flist_sort.o 2024-04-22T01:57:35.867 INFO:teuthology.orchestra.run.smithi152.stdout: CC lib/gauss.o 2024-04-22T01:57:35.957 INFO:teuthology.orchestra.run.smithi152.stdout: CC lib/getrusage.o 2024-04-22T01:57:35.991 INFO:teuthology.orchestra.run.smithi152.stdout: CC lib/hweight.o 2024-04-22T01:57:36.030 INFO:teuthology.orchestra.run.smithi152.stdout: CC lib/ieee754.o 2024-04-22T01:57:36.082 INFO:teuthology.orchestra.run.smithi152.stdout: CC lib/lfsr.o 2024-04-22T01:57:36.181 INFO:teuthology.orchestra.run.smithi152.stdout: CC lib/memalign.o 2024-04-22T01:57:36.225 INFO:teuthology.orchestra.run.smithi152.stdout: CC lib/memcpy.o 2024-04-22T01:57:36.369 INFO:teuthology.orchestra.run.smithi152.stdout: CC lib/mountcheck.o 2024-04-22T01:57:36.410 INFO:teuthology.orchestra.run.smithi152.stdout: CC lib/num2str.o 2024-04-22T01:57:36.477 INFO:teuthology.orchestra.run.smithi152.stdout: CC lib/output_buffer.o 2024-04-22T01:57:36.524 INFO:teuthology.orchestra.run.smithi152.stdout: CC lib/pattern.o 2024-04-22T01:57:36.668 INFO:teuthology.orchestra.run.smithi152.stdout: CC lib/prio_tree.o 2024-04-22T01:57:36.791 INFO:teuthology.orchestra.run.smithi152.stdout: CC lib/rand.o 2024-04-22T01:57:36.984 INFO:teuthology.orchestra.run.smithi152.stdout: CC lib/rbtree.o 2024-04-22T01:57:37.109 INFO:teuthology.orchestra.run.smithi152.stdout: CC lib/strntol.o 2024-04-22T01:57:37.156 INFO:teuthology.orchestra.run.smithi152.stdout: CC lib/zipf.o 2024-04-22T01:57:37.278 INFO:teuthology.orchestra.run.smithi152.stdout: CC gettime.o 2024-04-22T01:57:37.566 INFO:teuthology.orchestra.run.smithi152.stdout: CC ioengines.o 2024-04-22T01:57:37.828 INFO:teuthology.orchestra.run.smithi152.stdout: CC init.o 2024-04-22T01:57:38.607 INFO:teuthology.orchestra.run.smithi152.stdout: CC stat.o 2024-04-22T01:57:40.482 INFO:teuthology.orchestra.run.smithi152.stdout: CC log.o 2024-04-22T01:57:40.604 INFO:teuthology.orchestra.run.smithi152.stdout: CC time.o 2024-04-22T01:57:40.733 INFO:teuthology.orchestra.run.smithi152.stdout: CC filesetup.o 2024-04-22T01:57:41.437 INFO:teuthology.orchestra.run.smithi152.stdout: CC eta.o 2024-04-22T01:57:41.761 INFO:teuthology.orchestra.run.smithi152.stdout: CC verify.o 2024-04-22T01:57:42.446 INFO:teuthology.orchestra.run.smithi152.stdout: CC memory.o 2024-04-22T01:57:42.617 INFO:teuthology.orchestra.run.smithi152.stdout: CC io_u.o 2024-04-22T01:57:43.406 INFO:teuthology.orchestra.run.smithi152.stdout: CC parse.o 2024-04-22T01:57:43.858 INFO:teuthology.orchestra.run.smithi152.stdout: CC fio_sem.o 2024-04-22T01:57:43.962 INFO:teuthology.orchestra.run.smithi152.stdout: CC rwlock.o 2024-04-22T01:57:44.044 INFO:teuthology.orchestra.run.smithi152.stdout: CC pshared.o 2024-04-22T01:57:44.110 INFO:teuthology.orchestra.run.smithi152.stdout: CC options.o 2024-04-22T01:57:44.907 INFO:teuthology.orchestra.run.smithi152.stdout: CC smalloc.o 2024-04-22T01:57:45.092 INFO:teuthology.orchestra.run.smithi152.stdout: CC filehash.o 2024-04-22T01:57:45.246 INFO:teuthology.orchestra.run.smithi152.stdout: CC profile.o 2024-04-22T01:57:45.365 INFO:teuthology.orchestra.run.smithi152.stdout: CC debug.o 2024-04-22T01:57:45.406 INFO:teuthology.orchestra.run.smithi152.stdout: CC engines/cpu.o 2024-04-22T01:57:45.546 INFO:teuthology.orchestra.run.smithi152.stdout: CC engines/mmap.o 2024-04-22T01:57:45.711 INFO:teuthology.orchestra.run.smithi152.stdout: CC engines/sync.o 2024-04-22T01:57:45.942 INFO:teuthology.orchestra.run.smithi152.stdout: CC engines/null.o 2024-04-22T01:57:46.071 INFO:teuthology.orchestra.run.smithi152.stdout: CC engines/net.o 2024-04-22T01:57:46.504 INFO:teuthology.orchestra.run.smithi152.stdout: CC engines/ftruncate.o 2024-04-22T01:57:46.603 INFO:teuthology.orchestra.run.smithi152.stdout: CC engines/fileoperations.o 2024-04-22T01:57:46.764 INFO:teuthology.orchestra.run.smithi152.stdout: CC engines/exec.o 2024-04-22T01:57:46.917 INFO:teuthology.orchestra.run.smithi152.stdout: CC server.o 2024-04-22T01:57:47.657 INFO:teuthology.orchestra.run.smithi152.stdout: CC client.o 2024-04-22T01:57:48.200 INFO:teuthology.orchestra.run.smithi152.stdout: CC iolog.o 2024-04-22T01:57:48.645 INFO:teuthology.orchestra.run.smithi152.stdout: CC backend.o 2024-04-22T01:57:49.501 INFO:teuthology.orchestra.run.smithi152.stdout: CC libfio.o 2024-04-22T01:57:49.673 INFO:teuthology.orchestra.run.smithi152.stdout: CC flow.o 2024-04-22T01:57:49.793 INFO:teuthology.orchestra.run.smithi152.stdout: CC cconv.o 2024-04-22T01:57:50.149 INFO:teuthology.orchestra.run.smithi152.stdout: CC gettime-thread.o 2024-04-22T01:57:50.253 INFO:teuthology.orchestra.run.smithi152.stdout: CC helpers.o 2024-04-22T01:57:50.315 INFO:teuthology.orchestra.run.smithi152.stdout: CC json.o 2024-04-22T01:57:50.501 INFO:teuthology.orchestra.run.smithi152.stdout: CC idletime.o 2024-04-22T01:57:50.725 INFO:teuthology.orchestra.run.smithi152.stdout: CC td_error.o 2024-04-22T01:57:50.828 INFO:teuthology.orchestra.run.smithi152.stdout: CC profiles/tiobench.o 2024-04-22T01:57:50.932 INFO:teuthology.orchestra.run.smithi152.stdout: CC profiles/act.o 2024-04-22T01:57:51.091 INFO:teuthology.orchestra.run.smithi152.stdout: CC io_u_queue.o 2024-04-22T01:57:51.141 INFO:teuthology.orchestra.run.smithi152.stdout: CC filelock.o 2024-04-22T01:57:51.267 INFO:teuthology.orchestra.run.smithi152.stdout: CC workqueue.o 2024-04-22T01:57:51.442 INFO:teuthology.orchestra.run.smithi152.stdout: CC rate-submit.o 2024-04-22T01:57:51.592 INFO:teuthology.orchestra.run.smithi152.stdout: CC optgroup.o 2024-04-22T01:57:51.637 INFO:teuthology.orchestra.run.smithi152.stdout: CC helper_thread.o 2024-04-22T01:57:51.815 INFO:teuthology.orchestra.run.smithi152.stdout: CC steadystate.o 2024-04-22T01:57:52.056 INFO:teuthology.orchestra.run.smithi152.stdout: CC zone-dist.o 2024-04-22T01:57:52.189 INFO:teuthology.orchestra.run.smithi152.stdout: CC zbd.o 2024-04-22T01:57:52.896 INFO:teuthology.orchestra.run.smithi152.stdout: CC dedupe.o 2024-04-22T01:57:53.043 INFO:teuthology.orchestra.run.smithi152.stdout: CC fdp.o 2024-04-22T01:57:53.184 INFO:teuthology.orchestra.run.smithi152.stdout: CC engines/posixaio.o 2024-04-22T01:57:53.318 INFO:teuthology.orchestra.run.smithi152.stdout: CC engines/falloc.o 2024-04-22T01:57:53.430 INFO:teuthology.orchestra.run.smithi152.stdout: CC engines/e4defrag.o 2024-04-22T01:57:53.564 INFO:teuthology.orchestra.run.smithi152.stdout: CC engines/splice.o 2024-04-22T01:57:53.752 INFO:teuthology.orchestra.run.smithi152.stdout: CC oslib/asprintf.o 2024-04-22T01:57:53.788 INFO:teuthology.orchestra.run.smithi152.stdout: CC oslib/strlcat.o 2024-04-22T01:57:53.831 INFO:teuthology.orchestra.run.smithi152.stdout: CC engines/mtd.o 2024-04-22T01:57:54.003 INFO:teuthology.orchestra.run.smithi152.stdout: CC oslib/libmtd.o 2024-04-22T01:57:54.388 INFO:teuthology.orchestra.run.smithi152.stdout: CC oslib/libmtd_legacy.o 2024-04-22T01:57:54.524 INFO:teuthology.orchestra.run.smithi152.stdout: CC diskutil.o 2024-04-22T01:57:54.736 INFO:teuthology.orchestra.run.smithi152.stdout: CC fifo.o 2024-04-22T01:57:54.791 INFO:teuthology.orchestra.run.smithi152.stdout: CC blktrace.o 2024-04-22T01:57:55.131 INFO:teuthology.orchestra.run.smithi152.stdout: CC cgroup.o 2024-04-22T01:57:55.285 INFO:teuthology.orchestra.run.smithi152.stdout: CC trim.o 2024-04-22T01:57:55.398 INFO:teuthology.orchestra.run.smithi152.stdout: CC engines/sg.o 2024-04-22T01:57:55.788 INFO:teuthology.orchestra.run.smithi152.stdout: CC oslib/linux-dev-lookup.o 2024-04-22T01:57:55.847 INFO:teuthology.orchestra.run.smithi152.stdout: CC engines/io_uring.o 2024-04-22T01:57:56.237 INFO:teuthology.orchestra.run.smithi152.stdout: CC engines/nvme.o 2024-04-22T01:57:56.552 INFO:teuthology.orchestra.run.smithi152.stdout: CC oslib/linux-blkzoned.o 2024-04-22T01:57:56.705 INFO:teuthology.orchestra.run.smithi152.stdout: CC engines/cmdprio.o 2024-04-22T01:57:56.950 INFO:teuthology.orchestra.run.smithi152.stdout: CC engines/rados.o 2024-04-22T01:57:57.127 INFO:teuthology.orchestra.run.smithi152.stdout: CC engines/rbd.o 2024-04-22T01:57:57.311 INFO:teuthology.orchestra.run.smithi152.stdout: CC fio.o 2024-04-22T01:57:57.400 INFO:teuthology.orchestra.run.smithi152.stdout: LINK fio 2024-04-22T01:57:57.648 INFO:teuthology.orchestra.run.smithi152.stdout: CC t/log.o 2024-04-22T01:57:57.691 INFO:teuthology.orchestra.run.smithi152.stdout: CC t/genzipf.o 2024-04-22T01:57:57.829 INFO:teuthology.orchestra.run.smithi152.stdout: CC oslib/strcasestr.o 2024-04-22T01:57:57.853 INFO:teuthology.orchestra.run.smithi152.stdout: CC oslib/strndup.o 2024-04-22T01:57:57.877 INFO:teuthology.orchestra.run.smithi152.stdout: LINK t/fio-genzipf 2024-04-22T01:57:58.021 INFO:teuthology.orchestra.run.smithi152.stdout: CC t/btrace2fio.o 2024-04-22T01:57:58.441 INFO:teuthology.orchestra.run.smithi152.stdout: LINK t/fio-btrace2fio 2024-04-22T01:57:58.581 INFO:teuthology.orchestra.run.smithi152.stdout: CC t/verify-state.o 2024-04-22T01:57:58.680 INFO:teuthology.orchestra.run.smithi152.stdout: CC t/debug.o 2024-04-22T01:57:58.714 INFO:teuthology.orchestra.run.smithi152.stdout: LINK t/fio-verify-state 2024-04-22T01:57:58.855 INFO:teuthology.orchestra.run.smithi152.stdout: CC t/stest.o 2024-04-22T01:57:58.927 INFO:teuthology.orchestra.run.smithi152.stdout: CC t/arch.o 2024-04-22T01:57:58.957 INFO:teuthology.orchestra.run.smithi152.stdout: LINK t/stest 2024-04-22T01:57:59.098 INFO:teuthology.orchestra.run.smithi152.stdout: CC t/ieee754.o 2024-04-22T01:57:59.138 INFO:teuthology.orchestra.run.smithi152.stdout: LINK t/ieee754 2024-04-22T01:57:59.275 INFO:teuthology.orchestra.run.smithi152.stdout: CC t/axmap.o 2024-04-22T01:57:59.374 INFO:teuthology.orchestra.run.smithi152.stdout: LINK t/axmap 2024-04-22T01:57:59.514 INFO:teuthology.orchestra.run.smithi152.stdout: CC t/lfsr-test.o 2024-04-22T01:57:59.614 INFO:teuthology.orchestra.run.smithi152.stdout: LINK t/lfsr-test 2024-04-22T01:57:59.756 INFO:teuthology.orchestra.run.smithi152.stdout: CC t/gen-rand.o 2024-04-22T01:57:59.851 INFO:teuthology.orchestra.run.smithi152.stdout: LINK t/gen-rand 2024-04-22T01:57:59.994 INFO:teuthology.orchestra.run.smithi152.stdout: CC t/memlock.o 2024-04-22T01:58:00.054 INFO:teuthology.orchestra.run.smithi152.stdout: LINK t/memlock 2024-04-22T01:58:00.191 INFO:teuthology.orchestra.run.smithi152.stdout: CC t/read-to-pipe-async.o 2024-04-22T01:58:00.381 INFO:teuthology.orchestra.run.smithi152.stdout: LINK t/read-to-pipe-async 2024-04-22T01:58:00.520 INFO:teuthology.orchestra.run.smithi152.stdout: CC t/io_uring.o 2024-04-22T01:58:01.058 INFO:teuthology.orchestra.run.smithi152.stdout: LINK t/io_uring 2024-04-22T01:58:01.198 DEBUG:teuthology.orchestra.run.smithi152:> sudo /sbin/sysctl -q -w kernel.perf_event_paranoid=0 2024-04-22T01:58:01.222 DEBUG:teuthology.orchestra.run.smithi152:> /home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml 2024-04-22T01:58:01.446 INFO:teuthology.orchestra.run.smithi152.stderr:01:58:01 - DEBUG - cbt - Settings.cluster: 2024-04-22T01:58:01.446 INFO:teuthology.orchestra.run.smithi152.stderr: {'archive_dir': '/home/ubuntu/cephtest/archive/cbt', 2024-04-22T01:58:01.446 INFO:teuthology.orchestra.run.smithi152.stderr: 'clients': ['ubuntu@smithi152.front.sepia.ceph.com'], 2024-04-22T01:58:01.446 INFO:teuthology.orchestra.run.smithi152.stderr: 'conf_file': '/etc/ceph/ceph.conf', 2024-04-22T01:58:01.447 INFO:teuthology.orchestra.run.smithi152.stderr: 'head': 'ubuntu@smithi152.front.sepia.ceph.com', 2024-04-22T01:58:01.447 INFO:teuthology.orchestra.run.smithi152.stderr: 'is_teuthology': True, 2024-04-22T01:58:01.447 INFO:teuthology.orchestra.run.smithi152.stderr: 'iterations': 1, 2024-04-22T01:58:01.447 INFO:teuthology.orchestra.run.smithi152.stderr: 'mons': ['ubuntu@smithi152.front.sepia.ceph.com'], 2024-04-22T01:58:01.447 INFO:teuthology.orchestra.run.smithi152.stderr: 'osds': ['ubuntu@smithi152.front.sepia.ceph.com'], 2024-04-22T01:58:01.447 INFO:teuthology.orchestra.run.smithi152.stderr: 'osds_per_node': 3, 2024-04-22T01:58:01.447 INFO:teuthology.orchestra.run.smithi152.stderr: 'pid_dir': '/var/run/ceph', 2024-04-22T01:58:01.447 INFO:teuthology.orchestra.run.smithi152.stderr: 'pool_profiles': {'rbd': {'pg_size': 128, 'pgp_size': 128, 'replication': 3}}, 2024-04-22T01:58:01.448 INFO:teuthology.orchestra.run.smithi152.stderr: 'rebuild_every_test': False, 2024-04-22T01:58:01.448 INFO:teuthology.orchestra.run.smithi152.stderr: 'rgws': {'ubuntu@smithi152.front.sepia.ceph.com': None}, 2024-04-22T01:58:01.448 INFO:teuthology.orchestra.run.smithi152.stderr: 'tmp_dir': '/tmp/cbt', 2024-04-22T01:58:01.448 INFO:teuthology.orchestra.run.smithi152.stderr: 'use_existing': True, 2024-04-22T01:58:01.448 INFO:teuthology.orchestra.run.smithi152.stderr: 'user': 'ubuntu'} 2024-04-22T01:58:01.449 INFO:teuthology.orchestra.run.smithi152.stderr:01:58:01 - DEBUG - cbt - Nodes : ubuntu@smithi152.front.sepia.ceph.com 2024-04-22T01:58:01.449 INFO:teuthology.orchestra.run.smithi152.stderr:01:58:01 - DEBUG - cbt - Nodes : ubuntu@smithi152.front.sepia.ceph.com 2024-04-22T01:58:01.449 INFO:teuthology.orchestra.run.smithi152.stderr:01:58:01 - DEBUG - cbt - get_fqdn_local()=smithi152 2024-04-22T01:58:01.449 INFO:teuthology.orchestra.run.smithi152.stderr:01:58:01 - DEBUG - cbt - CheckedPopen continue_if_error=False, shell=False args=pdsh -S -f 1 -R ssh -w ubuntu@smithi152.front.sepia.ceph.com if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw; fi 2024-04-22T01:58:01.567 INFO:teuthology.orchestra.run.smithi152.stdout:cleaning remote dir /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw 2024-04-22T01:58:01.568 INFO:teuthology.orchestra.run.smithi152.stderr:Traceback (most recent call last): 2024-04-22T01:58:01.568 INFO:teuthology.orchestra.run.smithi152.stderr: File "/home/ubuntu/cephtest/cbt/cbt.py", line 93, in 2024-04-22T01:58:01.569 INFO:teuthology.orchestra.run.smithi152.stderr: exit(main(sys.argv)) 2024-04-22T01:58:01.569 INFO:teuthology.orchestra.run.smithi152.stderr: File "/home/ubuntu/cephtest/cbt/cbt.py", line 64, in main 2024-04-22T01:58:01.569 INFO:teuthology.orchestra.run.smithi152.stderr: b.initialize() 2024-04-22T01:58:01.569 INFO:teuthology.orchestra.run.smithi152.stderr: File "/home/ubuntu/cephtest/cbt/benchmark/librbdfio.py", line 73, in initialize 2024-04-22T01:58:01.569 INFO:teuthology.orchestra.run.smithi152.stderr: common.clean_remote_dir(self.run_dir) 2024-04-22T01:58:01.569 INFO:teuthology.orchestra.run.smithi152.stderr: File "/home/ubuntu/cephtest/cbt/common.py", line 232, in clean_remote_dir 2024-04-22T01:58:01.569 INFO:teuthology.orchestra.run.smithi152.stderr: pdsh(nodes, 'if [ -d "%s" ]; then rm -rf %s; fi' % (remote_dir, remote_dir), 2024-04-22T01:58:01.569 INFO:teuthology.orchestra.run.smithi152.stderr: File "/home/ubuntu/cephtest/cbt/common.py", line 55, in communicate 2024-04-22T01:58:01.569 INFO:teuthology.orchestra.run.smithi152.stderr: raise Exception('\n'.join([str(self), 2024-04-22T01:58:01.569 INFO:teuthology.orchestra.run.smithi152.stderr:Exception: checked_Popen args=pdsh -S -f 1 -R ssh -w ubuntu@smithi152.front.sepia.ceph.com if [ -d "/tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw" ]; then rm -rf /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-04194304/concurrent_procs-002/iodepth-032/randrw; fi continue_if_error=False rtncode=255 2024-04-22T01:58:01.569 INFO:teuthology.orchestra.run.smithi152.stderr:stdout: 2024-04-22T01:58:01.570 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-04-22T01:58:01.570 INFO:teuthology.orchestra.run.smithi152.stderr:stderr: 2024-04-22T01:58:01.570 INFO:teuthology.orchestra.run.smithi152.stderr:smithi152: Warning: Permanently added 'smithi152.front.sepia.ceph.com' (ECDSA) to the list of known hosts. 2024-04-22T01:58:01.570 INFO:teuthology.orchestra.run.smithi152.stderr:smithi152: Permission denied, please try again. 2024-04-22T01:58:01.570 INFO:teuthology.orchestra.run.smithi152.stderr:smithi152: Permission denied, please try again. 2024-04-22T01:58:01.570 INFO:teuthology.orchestra.run.smithi152.stderr:smithi152: ubuntu@smithi152.front.sepia.ceph.com: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). 2024-04-22T01:58:01.570 INFO:teuthology.orchestra.run.smithi152.stderr:pdsh@smithi152: smithi152: ssh exited with exit code 255 2024-04-22T01:58:01.570 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-04-22T01:58:01.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:58:01.580 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/cbt.py", line 244, in begin self.first_mon.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi152 with status 1: '/home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml' 2024-04-22T01:58:01.789 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=fd22be17c0cf40e090451501e5d7c3f9 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_676947e1f73a9d7f900df5a6854c283c4eef24a9/qa/tasks/cbt.py", line 244, in begin self.first_mon.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi152 with status 1: '/home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml' 2024-04-22T01:58:01.791 DEBUG:teuthology.run_tasks:Unwinding manager cbt 2024-04-22T01:58:01.800 DEBUG:teuthology.orchestra.run.smithi152:> rm --one-file-system -rf -- /home/ubuntu/cephtest/cbt 2024-04-22T01:58:01.823 DEBUG:teuthology.orchestra.run.smithi152:> rm --one-file-system -rf -- /home/ubuntu/cephtest/fio 2024-04-22T01:58:01.895 INFO:tasks.cbt_performance:Collecting CBT performance data config 2024-04-22T01:58:01.895 INFO:tasks.cbt_performance:reading cbt preferences 2024-04-22T01:58:01.896 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest -name 'json_output.*' 2024-04-22T01:58:01.950 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-22T01:58:01.959 DEBUG:teuthology.orchestra.run.smithi152:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-22T01:58:02.104 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:58:02.099+0000 7fe696d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:02.106 INFO:teuthology.orchestra.run.smithi152.stderr:2024-04-22T01:58:02.108+0000 7fe696d5d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2024-04-22T01:58:02.313 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-22T01:58:02.313 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-22T01:58:02.314 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-22T01:58:02.314 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:58:02.372 INFO:tasks.ceph.osd.0:Stopped 2024-04-22T01:58:02.372 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-22T01:58:02.372 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:58:02.436 INFO:tasks.ceph.osd.1:Stopped 2024-04-22T01:58:02.437 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-22T01:58:02.437 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:58:02.500 INFO:tasks.ceph.osd.2:Stopped 2024-04-22T01:58:02.501 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-22T01:58:02.501 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-22T01:58:02.501 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:58:02.527 INFO:tasks.ceph.mgr.x:Stopped 2024-04-22T01:58:02.527 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-22T01:58:02.527 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-22T01:58:02.527 INFO:teuthology.orchestra.run:waiting for 300 2024-04-22T01:58:02.577 INFO:tasks.ceph.mon.a:Stopped 2024-04-22T01:58:02.578 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-22T01:58:02.578 DEBUG:teuthology.orchestra.run.smithi152:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_' | egrep -v '\(OSD_' | egrep -v '\(OBJECT_' | egrep -v 'overall HEALTH' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-04-22T01:58:02.651 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi152.front.sepia.ceph.com 2024-04-22T01:58:02.651 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-22T01:58:02.885 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi152.front.sepia.ceph.com 2024-04-22T01:58:02.885 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-22T01:58:03.017 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi152.front.sepia.ceph.com 2024-04-22T01:58:03.017 DEBUG:teuthology.orchestra.run.smithi152:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-22T01:58:03.106 INFO:tasks.ceph:Archiving mon data... 2024-04-22T01:58:03.107 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668110/data/mon.a.tgz 2024-04-22T01:58:03.107 DEBUG:teuthology.orchestra.run.smithi152:> mktemp 2024-04-22T01:58:03.121 INFO:teuthology.orchestra.run.smithi152.stdout:/tmp/tmp.TqM3KHpV8g 2024-04-22T01:58:03.121 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.TqM3KHpV8g 2024-04-22T01:58:03.207 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0666 /tmp/tmp.TqM3KHpV8g 2024-04-22T01:58:03.288 DEBUG:teuthology.orchestra.remote:smithi152:/tmp/tmp.TqM3KHpV8g is 138KB 2024-04-22T01:58:03.340 DEBUG:teuthology.orchestra.run.smithi152:> rm -fr /tmp/tmp.TqM3KHpV8g 2024-04-22T01:58:03.354 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-22T01:58:03.354 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-22T01:58:03.473 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-22T01:58:03.473 INFO:tasks.ceph:Archiving crash dumps... 2024-04-22T01:58:03.475 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/crash to /home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668110/remote/smithi152/crash 2024-04-22T01:58:03.475 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-22T01:58:03.536 INFO:tasks.ceph:Compressing logs... 2024-04-22T01:58:03.537 DEBUG:teuthology.orchestra.run.smithi152:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T01:58:03.599 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29353.log 2024-04-22T01:58:03.615 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28636.log 2024-04-22T01:58:03.615 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.29353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27886.log 2024-04-22T01:58:03.615 INFO:teuthology.orchestra.run.smithi152.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29353.log.gz 2024-04-22T01:58:03.616 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.28636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27728.log 2024-04-22T01:58:03.616 INFO:teuthology.orchestra.run.smithi152.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28636.log.gz 2024-04-22T01:58:03.616 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.27886.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-22T01:58:03.616 INFO:teuthology.orchestra.run.smithi152.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27886.log.gz 2024-04-22T01:58:03.616 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.27728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28723.log 2024-04-22T01:58:03.616 INFO:teuthology.orchestra.run.smithi152.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.27728.log.gz 2024-04-22T01:58:03.616 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-22T01:58:03.616 INFO:teuthology.orchestra.run.smithi152.stderr: 83.0%/var/log/ceph/ceph-client.admin.28723.log: -- replaced with /var/log/ceph/ceph.log.gz 2024-04-22T01:58:03.617 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28888.log 2024-04-22T01:58:03.617 INFO:teuthology.orchestra.run.smithi152.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28723.log.gz 2024-04-22T01:58:03.617 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28763.log 2024-04-22T01:58:03.617 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.28888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27527.log 2024-04-22T01:58:03.617 INFO:teuthology.orchestra.run.smithi152.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28888.log.gz 2024-04-22T01:58:03.617 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.28763.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-22T01:58:03.617 INFO:teuthology.orchestra.run.smithi152.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28763.log.gz 2024-04-22T01:58:03.617 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.27527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28513.log 2024-04-22T01:58:03.618 INFO:teuthology.orchestra.run.smithi152.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27527.log.gz 2024-04-22T01:58:03.618 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28635.log 2024-04-22T01:58:03.618 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.28513.log: 86.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-22T01:58:03.618 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28471.log 2024-04-22T01:58:03.618 INFO:teuthology.orchestra.run.smithi152.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28513.log.gz 2024-04-22T01:58:03.618 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.28635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29140.log 2024-04-22T01:58:03.618 INFO:teuthology.orchestra.run.smithi152.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28635.log.gz 2024-04-22T01:58:03.618 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.28471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28889.log 2024-04-22T01:58:03.619 INFO:teuthology.orchestra.run.smithi152.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28471.log.gz 2024-04-22T01:58:03.619 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.29140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29141.log 2024-04-22T01:58:03.619 INFO:teuthology.orchestra.run.smithi152.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29140.log.gz 2024-04-22T01:58:03.619 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.28889.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.26080.log 2024-04-22T01:58:03.619 INFO:teuthology.orchestra.run.smithi152.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28889.log.gz 2024-04-22T01:58:03.619 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.29141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29101.log 2024-04-22T01:58:03.619 INFO:teuthology.orchestra.run.smithi152.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29141.log.gz 2024-04-22T01:58:03.619 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph.tmp-client.admin.26080.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-22T01:58:03.619 INFO:teuthology.orchestra.run.smithi152.stderr: 68.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.26080.log.gz 2024-04-22T01:58:03.620 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.29101.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-22T01:58:03.620 INFO:teuthology.orchestra.run.smithi152.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29101.log.gz 2024-04-22T01:58:03.620 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28368.log 2024-04-22T01:58:03.620 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-22T01:58:03.620 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.28368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28975.log 2024-04-22T01:58:03.620 INFO:teuthology.orchestra.run.smithi152.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28368.log.gz 2024-04-22T01:58:03.620 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28762.log 2024-04-22T01:58:03.620 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.28975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29014.log 2024-04-22T01:58:03.620 INFO:teuthology.orchestra.run.smithi152.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.28975.log.gz 2024-04-22T01:58:03.621 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28849.log 2024-04-22T01:58:03.621 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.28762.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28762.log.gz 2024-04-22T01:58:03.621 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.29014.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-22T01:58:03.621 INFO:teuthology.orchestra.run.smithi152.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29014.log.gz 2024-04-22T01:58:03.621 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.28849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29395.log 2024-04-22T01:58:03.621 INFO:teuthology.orchestra.run.smithi152.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.28849.log.gz 2024-04-22T01:58:03.621 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28555.log 2024-04-22T01:58:03.621 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.29395.log: 50.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27688.log 2024-04-22T01:58:03.621 INFO:teuthology.orchestra.run.smithi152.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29395.log.gz 2024-04-22T01:58:03.622 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.28555.log: gzip 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz -5 2024-04-22T01:58:03.622 INFO:teuthology.orchestra.run.smithi152.stderr: --verbose -- /var/log/ceph/ceph-client.admin.27916.log 2024-04-22T01:58:03.622 INFO:teuthology.orchestra.run.smithi152.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28555.log.gz 2024-04-22T01:58:03.622 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29227.log 2024-04-22T01:58:03.622 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.27688.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27688.log.gz 2024-04-22T01:58:03.622 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.27916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27439.log 2024-04-22T01:58:03.623 INFO:teuthology.orchestra.run.smithi152.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27916.log.gz 2024-04-22T01:58:03.623 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29247.log 2024-04-22T01:58:03.623 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.29227.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29227.log.gz 2024-04-22T01:58:03.623 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.27439.log: gzip -5 50.4% -- replaced with /var/log/ceph/ceph-client.admin.27439.log.gz 2024-04-22T01:58:03.623 INFO:teuthology.orchestra.run.smithi152.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29015.log 2024-04-22T01:58:03.623 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.29247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28410.log 2024-04-22T01:58:03.624 INFO:teuthology.orchestra.run.smithi152.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.29247.log.gz 2024-04-22T01:58:03.624 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.29015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32847.log 2024-04-22T01:58:03.624 INFO:teuthology.orchestra.run.smithi152.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.29015.log.gz 2024-04-22T01:58:03.624 INFO:teuthology.orchestra.run.smithi152.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29268.log 2024-04-22T01:58:03.624 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.28410.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.28410.log.gz 2024-04-22T01:58:03.624 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.32847.log: gzip -5 --verbose 49.6% -- replaced with /var/log/ceph/ceph-client.admin.32847.log.gz 2024-04-22T01:58:03.624 INFO:teuthology.orchestra.run.smithi152.stderr: -- /var/log/ceph/ceph-client.admin.27647.log 2024-04-22T01:58:03.625 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.29268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29437.log 2024-04-22T01:58:03.625 INFO:teuthology.orchestra.run.smithi152.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29268.log.gz 2024-04-22T01:58:03.625 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.27647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28312.log 2024-04-22T01:58:03.625 INFO:teuthology.orchestra.run.smithi152.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.27647.log.gz 2024-04-22T01:58:03.625 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.29437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28637.log 2024-04-22T01:58:03.625 INFO:teuthology.orchestra.run.smithi152.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.29437.log.gz 2024-04-22T01:58:03.625 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.28312.log: 50.8% -- replaced with /var/log/ceph/ceph-client.admin.28312.log.gz 2024-04-22T01:58:03.626 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/ceph-client.admin.28637.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.28637.log.gz 2024-04-22T01:58:03.626 INFO:teuthology.orchestra.run.smithi152.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-22T01:58:03.626 INFO:teuthology.orchestra.run.smithi152.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-22T01:58:03.629 INFO:teuthology.orchestra.run.smithi152.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-22T01:58:03.630 INFO:teuthology.orchestra.run.smithi152.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-22T01:58:03.632 INFO:teuthology.orchestra.run.smithi152.stderr: 2024-04-22T01:58:03.632 INFO:teuthology.orchestra.run.smithi152.stderr:real 0m0.041s 2024-04-22T01:58:03.632 INFO:teuthology.orchestra.run.smithi152.stderr:user 0m0.089s 2024-04-22T01:58:03.632 INFO:teuthology.orchestra.run.smithi152.stderr:sys 0m0.032s 2024-04-22T01:58:03.632 INFO:tasks.ceph:Archiving logs... 2024-04-22T01:58:03.632 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/log/ceph to /home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668110/remote/smithi152/log 2024-04-22T01:58:03.633 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T01:58:03.776 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-22T01:58:03.787 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-22T01:58:03.788 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-22T01:58:03.828 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-22T01:58:03.837 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-22T01:58:03.838 DEBUG:teuthology.orchestra.run.smithi152:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T01:58:03.865 INFO:teuthology.orchestra.run.smithi152.stderr:bash: line 1: ntpq: command not found 2024-04-22T01:58:03.870 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T01:58:03.870 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2024-04-22T01:58:03.870 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 1 +50us[ +50us] +/- 95ms 2024-04-22T01:58:03.870 INFO:teuthology.orchestra.run.smithi152.stdout:^* hv02.front.sepia.ceph.com 3 6 377 1 -378us[ -374us] +/- 65ms 2024-04-22T01:58:03.870 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 0 +381us[ +381us] +/- 69ms 2024-04-22T01:58:03.870 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T01:58:03.871 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-22T01:58:03.880 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-22T01:58:03.881 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-22T01:58:03.897 DEBUG:teuthology.orchestra.run.smithi152:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T01:58:03.968 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-22T01:58:03.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:58:03.994 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-22T01:58:04.003 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-22T01:58:04.012 INFO:teuthology.task.internal:Duration was 473.261920 seconds 2024-04-22T01:58:04.012 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-22T01:58:04.021 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-22T01:58:04.021 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T01:58:04.070 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T01:58:04.494 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-22T01:58:04.494 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi152.front.sepia.ceph.com 2024-04-22T01:58:04.494 DEBUG:teuthology.orchestra.run.smithi152:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T01:58:04.554 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-22T01:58:04.554 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T01:58:04.619 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-22T01:58:04.620 DEBUG:teuthology.orchestra.run.smithi152:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T01:58:04.752 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-22T01:58:04.763 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-22T01:58:04.763 DEBUG:teuthology.orchestra.run.smithi152:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T01:58:04.787 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-22T01:58:04.797 DEBUG:teuthology.orchestra.run.smithi152:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T01:58:04.849 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = core 2024-04-22T01:58:04.886 DEBUG:teuthology.orchestra.run.smithi152:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T01:58:04.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T01:58:04.940 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-22T01:58:04.950 INFO:teuthology.task.internal:Transferring archived files... 2024-04-22T01:58:04.951 DEBUG:teuthology.misc:Transferring archived files from smithi152:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668110/remote/smithi152 2024-04-22T01:58:04.951 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T01:58:05.015 INFO:teuthology.task.internal:Removing archive directory... 2024-04-22T01:58:05.016 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T01:58:05.062 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-22T01:58:05.074 INFO:teuthology.task.internal:Not uploading archives. 2024-04-22T01:58:05.074 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-22T01:58:05.083 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-22T01:58:05.084 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T01:58:05.116 INFO:teuthology.orchestra.run.smithi152.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 01:58 /home/ubuntu/cephtest 2024-04-22T01:58:05.117 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-22T01:58:05.128 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-22T01:58:05.150 INFO:teuthology.nuke:Checking targets against current locks 2024-04-22T01:58:05.177 DEBUG:teuthology.nuke:shortname: smithi152 2024-04-22T01:58:05.178 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T01:58:05.200 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yingxin-2024-04-22_01:37:40-crimson-rados-ci-yingxin-crimson-fix-tri-mutex-3-distro-default-smithi/7668110', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 01:44:23.175631', 'locked_by': 'scheduled_yingxin@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK7vEJZAHyaYpnTOgKeOB12u7qU8K1guXN09pp+HLdP6rPm6XeYpmk9p/CF/C2mX0lNi+GqDfPWdLPJ+2dbhBvU='} 2024-04-22T01:58:05.247 INFO:teuthology.orchestra.console.smithi152:Power off 2024-04-22T01:58:05.248 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T01:58:05.270 DEBUG:teuthology.orchestra.console.smithi152:power off output: Chassis Power Control: Down/Off 2024-04-22T01:58:05.270 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T01:58:05.294 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2024-04-22T01:58:09.295 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T01:58:09.405 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2024-04-22T01:58:13.406 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T01:58:13.516 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is off 2024-04-22T01:58:13.617 INFO:teuthology.orchestra.console.smithi152:Power off completed 2024-04-22T01:58:13.725 INFO:teuthology.run:Summary data: description: crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} duration: 473.26191997528076 failure_reason: 'Command failed on smithi152 with status 1: ''/home/ubuntu/cephtest/cbt/cbt.py -a /home/ubuntu/cephtest/archive/cbt /home/ubuntu/cephtest/archive/cbt/cbt_config.yaml''' flavor: crimson owner: scheduled_yingxin@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=fd22be17c0cf40e090451501e5d7c3f9 status: fail success: false 2024-04-22T01:58:13.725 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T01:58:13.802 INFO:teuthology.run:FAIL